[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:49:26 dist-test-slave-wzcb krb5kdc[9595](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:49:26 dist-test-slave-wzcb krb5kdc[9595](info): set up 2 sockets
Sep 28 16:49:26 dist-test-slave-wzcb krb5kdc[9595](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 16:49:27.422636 9588 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:49:27 dist-test-slave-wzcb krb5kdc[9595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:49:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:49:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:49:27.616953 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:42789
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:37579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:42789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:27.973922 9608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:27.974275 9608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:27.974380 9608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:27.982023 9608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:27.982164 9608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:27.982215 9608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:27.982277 9608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:27.982323 9608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:27.992991 9608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:37579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:42789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:42789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:27.994505 9608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:27.996053 9608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:28.009354 9616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:28.009483 9613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:28.009421 9614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:28.009699 9608 server_base.cc:1034] running on GCE node
I20260928 16:49:28.010860 9608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:28.012604 9608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:28.013885 9608 hybrid_clock.cc:648] HybridClock initialized: now 1790614168013821 us; error 43 us; skew 500 ppm
I20260928 16:49:28.014359 9608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:28 dist-test-slave-wzcb krb5kdc[9595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614168, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:28.017866 9608 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:49:28.019608 9608 webserver.cc:460] Webserver started at http://127.9.93.62:39603/ using document root <none> and password file <none>
I20260928 16:49:28.020414 9608 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:28.020553 9608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:28.020897 9608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:28.024758 9608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "6bf78e467c934f7d8def3f75bf4334a1"
format_stamp: "Formatted at 2026-09-28 16:49:28 on dist-test-slave-wzcb"
server_key: "e9eba017e882b218f7deb10d3b8e5788"
server_key_iv: "1380b2246c3e293c6998fd3219613d36"
server_key_version: "encryptionkey@0"
I20260928 16:49:28.025743 9608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "6bf78e467c934f7d8def3f75bf4334a1"
format_stamp: "Formatted at 2026-09-28 16:49:28 on dist-test-slave-wzcb"
server_key: "e9eba017e882b218f7deb10d3b8e5788"
server_key_iv: "1380b2246c3e293c6998fd3219613d36"
server_key_version: "encryptionkey@0"
I20260928 16:49:28.032423 9608 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 16:49:28.036839 9623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:28.038611 9608 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260928 16:49:28.038844 9608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "6bf78e467c934f7d8def3f75bf4334a1"
format_stamp: "Formatted at 2026-09-28 16:49:28 on dist-test-slave-wzcb"
server_key: "e9eba017e882b218f7deb10d3b8e5788"
server_key_iv: "1380b2246c3e293c6998fd3219613d36"
server_key_version: "encryptionkey@0"
I20260928 16:49:28.039080 9608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:28.064479 9608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:28.069286 9608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:28.069609 9608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:28.088479 9608 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:42789
I20260928 16:49:28.088507 9675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:42789 every 8 connection(s)
I20260928 16:49:28.090055 9608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:49:28.094630 9676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:28.097491 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 9608
I20260928 16:49:28.097831 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:49:28.100091 9588 external_mini_cluster.cc:1428] Setting key c3c18a3dc2a89832ddf49b2711a47da2
I20260928 16:49:28.108114 9676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf78e467c934f7d8def3f75bf4334a1: Bootstrap starting.
I20260928 16:49:28.111840 9676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bf78e467c934f7d8def3f75bf4334a1: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:28.113068 9676 log.cc:824] T 00000000000000000000000000000000 P 6bf78e467c934f7d8def3f75bf4334a1: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:28.116695 9676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf78e467c934f7d8def3f75bf4334a1: No bootstrap required, opened a new log
I20260928 16:49:28.124727 9676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bf78e467c934f7d8def3f75bf4334a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf78e467c934f7d8def3f75bf4334a1" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42789 } }
I20260928 16:49:28.125196 9676 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bf78e467c934f7d8def3f75bf4334a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:28.125311 9676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bf78e467c934f7d8def3f75bf4334a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bf78e467c934f7d8def3f75bf4334a1, State: Initialized, Role: FOLLOWER
I20260928 16:49:28.126024 9676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bf78e467c934f7d8def3f75bf4334a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf78e467c934f7d8def3f75bf4334a1" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42789 } }
I20260928 16:49:28.126295 9676 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bf78e467c934f7d8def3f75bf4334a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:49:28.126431 9676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bf78e467c934f7d8def3f75bf4334a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:49:28.126610 9676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bf78e467c934f7d8def3f75bf4334a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:28.129228 9676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bf78e467c934f7d8def3f75bf4334a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf78e467c934f7d8def3f75bf4334a1" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42789 } }
I20260928 16:49:28.129869 9676 leader_election.cc:302] T 00000000000000000000000000000000 P 6bf78e467c934f7d8def3f75bf4334a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bf78e467c934f7d8def3f75bf4334a1; no voters:
Sep 28 16:49:28 dist-test-slave-wzcb krb5kdc[9595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:49:28.130690 9676 leader_election.cc:288] T 00000000000000000000000000000000 P 6bf78e467c934f7d8def3f75bf4334a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:49:28.130836 9681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bf78e467c934f7d8def3f75bf4334a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:28.132040 9681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bf78e467c934f7d8def3f75bf4334a1 [term 1 LEADER]: Becoming Leader. State: Replica: 6bf78e467c934f7d8def3f75bf4334a1, State: Running, Role: LEADER
I20260928 16:49:28.132786 9681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bf78e467c934f7d8def3f75bf4334a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf78e467c934f7d8def3f75bf4334a1" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42789 } }
I20260928 16:49:28.133349 9676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bf78e467c934f7d8def3f75bf4334a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:28.136754 9683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf78e467c934f7d8def3f75bf4334a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bf78e467c934f7d8def3f75bf4334a1. Latest consensus state: current_term: 1 leader_uuid: "6bf78e467c934f7d8def3f75bf4334a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf78e467c934f7d8def3f75bf4334a1" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42789 } } }
I20260928 16:49:28.137115 9682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf78e467c934f7d8def3f75bf4334a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bf78e467c934f7d8def3f75bf4334a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf78e467c934f7d8def3f75bf4334a1" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42789 } } }
I20260928 16:49:28.138628 9682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf78e467c934f7d8def3f75bf4334a1 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:28.139215 9683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf78e467c934f7d8def3f75bf4334a1 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:28.147538 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:28.104757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51894 (local address 127.9.93.62:42789)
0928 16:49:28.105804 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:28.105823 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:28.107042 (+ 1219us) server_negotiation.cc:407] Connection header received
0928 16:49:28.116570 (+ 9528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:28.116599 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:28.117101 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:28.117574 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:28.121060 (+ 3486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:28.122574 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:28.124598 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:28.125449 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:28.125679 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:28.132661 (+ 6982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:28.132677 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:28.132703 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:28.132770 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:28.137463 (+ 4693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:28.138801 (+ 1338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:28.138809 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:28.138818 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:28.139624 (+ 806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:28.140200 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:28.140206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:28.140210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:28.142013 (+ 1803us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:28.142392 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:28.145318 (+ 2926us) server_negotiation.cc:299] Negotiation successful
0928 16:49:28.145939 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":473,"threads_started":1}
I20260928 16:49:28.149065 9688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:49:28.155982 9688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:49:28.167443 9688 catalog_manager.cc:1347] Generated new cluster ID: 865a3f0afca74af894d097583b7bc96b
I20260928 16:49:28.167603 9688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:49:28.183581 9688 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:49:28.185688 9688 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:49:28.205938 9688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bf78e467c934f7d8def3f75bf4334a1: Generated new TSK 0
I20260928 16:49:28.207294 9688 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
I20260928 16:49:28.255242 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:42789
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:37579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:28.619289 9702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:28.619697 9702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:28.619812 9702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:28.627761 9702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:28.627905 9702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:28.628031 9702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:49:28.638948 9702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:37579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:42789
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:28.640481 9702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:28.642030 9702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:28.657275 9710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:28.657353 9707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:28.657353 9708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:28.658536 9702 server_base.cc:1034] running on GCE node
I20260928 16:49:28.659335 9702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:28.660521 9702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:28.661742 9702 hybrid_clock.cc:648] HybridClock initialized: now 1790614168661670 us; error 73 us; skew 500 ppm
I20260928 16:49:28.662104 9702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:28 dist-test-slave-wzcb krb5kdc[9595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614168, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:28.665752 9702 init.cc:377] Logged in from keytab as kudu/127.9.93.1@KRBTEST.COM (short username kudu)
I20260928 16:49:28.667666 9702 webserver.cc:460] Webserver started at http://127.9.93.1:33257/ using document root <none> and password file <none>
I20260928 16:49:28.668509 9702 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:28.668634 9702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:28.668948 9702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:28.672540 9702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "151bf3ab3a4e4ac5b2ec4c0aeeb8fd5c"
format_stamp: "Formatted at 2026-09-28 16:49:28 on dist-test-slave-wzcb"
server_key: "952e0e58eb35d95c987370538b1d9460"
server_key_iv: "116ed571121d215c0074efbd361963c6"
server_key_version: "encryptionkey@0"
I20260928 16:49:28.673532 9702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "151bf3ab3a4e4ac5b2ec4c0aeeb8fd5c"
format_stamp: "Formatted at 2026-09-28 16:49:28 on dist-test-slave-wzcb"
server_key: "952e0e58eb35d95c987370538b1d9460"
server_key_iv: "116ed571121d215c0074efbd361963c6"
server_key_version: "encryptionkey@0"
I20260928 16:49:28.680263 9702 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260928 16:49:28.684790 9717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:28.686581 9702 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260928 16:49:28.686786 9702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "151bf3ab3a4e4ac5b2ec4c0aeeb8fd5c"
format_stamp: "Formatted at 2026-09-28 16:49:28 on dist-test-slave-wzcb"
server_key: "952e0e58eb35d95c987370538b1d9460"
server_key_iv: "116ed571121d215c0074efbd361963c6"
server_key_version: "encryptionkey@0"
I20260928 16:49:28.687011 9702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:28.705664 9702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:28.710891 9702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:28.711201 9702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:28.712423 9702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:49:28.714108 9702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:49:28.714206 9702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:28.714315 9702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:49:28.714363 9702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:28.751946 9702 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:45771
I20260928 16:49:28.751991 9829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:45771 every 8 connection(s)
I20260928 16:49:28.753489 9702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:49:28.755944 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 9702
I20260928 16:49:28.756210 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:49:28.757128 9588 external_mini_cluster.cc:1428] Setting key bf042472c11ff376b2595a79a137be4a
Sep 28 16:49:28 dist-test-slave-wzcb krb5kdc[9595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614168, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.2@KRBTEST.COM" created.
I20260928 16:49:28.778538 9832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:28.756727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:60211 (local address 127.9.93.62:42789)
0928 16:49:28.757268 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:28.757275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:28.758892 (+ 1617us) server_negotiation.cc:407] Connection header received
0928 16:49:28.761064 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:28.761071 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:28.761174 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:28.761303 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:28.763177 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:28.764252 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:28.766081 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:28.766779 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:28.766922 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:28.771469 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:28.771475 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:28.771479 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:28.771525 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:28.774352 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:28.775470 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:28.775477 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:28.775481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:28.775690 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:28.776295 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:28.776301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:28.776303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:28.776643 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:28.776796 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:28.777877 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0928 16:49:28.778197 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":274,"threads_started":1}
I20260928 16:49:28.780001 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:28.757623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42789 (local address 127.9.93.1:60211)
0928 16:49:28.758681 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:28.758743 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:28.760700 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:28.761560 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:28.761570 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:28.762252 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:28.762958 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:28.762980 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:28.764440 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:28.764446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:28.765885 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:28.765900 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:28.767037 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:28.767043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:28.767350 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:28.768118 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:28.768144 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:28.771078 (+ 2934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:28.774584 (+ 3506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:28.774596 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:28.774616 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:28.775195 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:28.775842 (+ 647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:28.775846 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:28.775849 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:28.776137 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:28.776809 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:28.776817 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:28.777325 (+ 508us) client_negotiation.cc:769] Sending connection context
0928 16:49:28.777639 (+ 314us) client_negotiation.cc:240] Negotiation successful
0928 16:49:28.778233 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":418,"threads_started":1}
I20260928 16:49:28.781967 9830 heartbeater.cc:347] Connected to a master server at 127.9.93.62:42789
I20260928 16:49:28.782366 9830 heartbeater.cc:464] Registering TS with master...
I20260928 16:49:28.783293 9830 heartbeater.cc:511] Master 127.9.93.62:42789 requested a full tablet report, sending...
I20260928 16:49:28.785889 9640 ts_manager.cc:194] Registered new tserver with Master: 151bf3ab3a4e4ac5b2ec4c0aeeb8fd5c (127.9.93.1:45771)
I20260928 16:49:28.787653 9640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.1@KRBTEST.COM'} at 127.9.93.1:60211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
I20260928 16:49:28.803989 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:42789
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:37579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:29.183915 9836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:29.184293 9836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:29.184453 9836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:29.192759 9836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:29.192934 9836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:29.193112 9836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:49:29.207556 9836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:37579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:42789
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:29.209723 9836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:29.211701 9836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:29.227335 9841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:29.227258 9844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:29.227290 9842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:29.228399 9836 server_base.cc:1034] running on GCE node
I20260928 16:49:29.229187 9836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:29.230399 9836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:29.231632 9836 hybrid_clock.cc:648] HybridClock initialized: now 1790614169231566 us; error 55 us; skew 500 ppm
I20260928 16:49:29.232014 9836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:29 dist-test-slave-wzcb krb5kdc[9595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614169, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:29.235837 9836 init.cc:377] Logged in from keytab as kudu/127.9.93.2@KRBTEST.COM (short username kudu)
I20260928 16:49:29.237735 9836 webserver.cc:460] Webserver started at http://127.9.93.2:42665/ using document root <none> and password file <none>
I20260928 16:49:29.238603 9836 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:29.238745 9836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:29.239097 9836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:29.242766 9836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "bd4e3db393974a93bbd64610622d81b5"
format_stamp: "Formatted at 2026-09-28 16:49:29 on dist-test-slave-wzcb"
server_key: "20bc3b76650848b044b1d38ff7255086"
server_key_iv: "1342fb6d6c04d94abdc0db62f96648ce"
server_key_version: "encryptionkey@0"
I20260928 16:49:29.243820 9836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "bd4e3db393974a93bbd64610622d81b5"
format_stamp: "Formatted at 2026-09-28 16:49:29 on dist-test-slave-wzcb"
server_key: "20bc3b76650848b044b1d38ff7255086"
server_key_iv: "1342fb6d6c04d94abdc0db62f96648ce"
server_key_version: "encryptionkey@0"
I20260928 16:49:29.250554 9836 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:49:29.255349 9851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:29.257201 9836 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260928 16:49:29.257411 9836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "bd4e3db393974a93bbd64610622d81b5"
format_stamp: "Formatted at 2026-09-28 16:49:29 on dist-test-slave-wzcb"
server_key: "20bc3b76650848b044b1d38ff7255086"
server_key_iv: "1342fb6d6c04d94abdc0db62f96648ce"
server_key_version: "encryptionkey@0"
I20260928 16:49:29.257658 9836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:29.297838 9836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:29.302721 9836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:29.303074 9836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:29.304323 9836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:49:29.306149 9836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:49:29.306247 9836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:29.306401 9836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:49:29.306461 9836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:29.342806 9836 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:35005
I20260928 16:49:29.342864 9963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:35005 every 8 connection(s)
I20260928 16:49:29.344501 9836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:49:29.351152 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 9836
I20260928 16:49:29.351410 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:49:29.352008 9588 external_mini_cluster.cc:1428] Setting key 0a96115c4f22629a6e9bf9a5dd0f7aac
Sep 28 16:49:29 dist-test-slave-wzcb krb5kdc[9595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614169, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.93.3@KRBTEST.COM; defaulting to no policy
I20260928 16:49:29.372737 9966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:29.347706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:56773 (local address 127.9.93.62:42789)
0928 16:49:29.348231 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:29.348238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:29.349652 (+ 1414us) server_negotiation.cc:407] Connection header received
0928 16:49:29.352033 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:29.352041 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:29.352183 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:29.352360 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:29.354907 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:29.356040 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:29.358199 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:29.358884 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:29.359045 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:29.364748 (+ 5703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:29.364755 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:29.364758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:29.364807 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:29.368050 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:29.369477 (+ 1427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:29.369482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:29.369484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:29.369693 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:29.370401 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:29.370405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:29.370407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:29.370775 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:29.370920 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:29.372095 (+ 1175us) server_negotiation.cc:299] Negotiation successful
0928 16:49:29.372359 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":246,"threads_started":1}
I20260928 16:49:29.373778 9967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:29.348525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42789 (local address 127.9.93.2:56773)
0928 16:49:29.349444 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:29.349495 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:29.351621 (+ 2126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:29.352823 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:29.352843 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:29.353774 (+ 931us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:29.354674 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:29.354702 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:29.356316 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:29.356321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:29.357954 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:29.357978 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:29.359240 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:29.359247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:29.359687 (+ 440us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:29.360543 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:29.360583 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:29.364288 (+ 3705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:29.368330 (+ 4042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:29.368345 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:29.368372 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:29.369274 (+ 902us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:29.369855 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:29.369861 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:29.369865 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:29.370234 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:29.370949 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:29.370959 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:29.371587 (+ 628us) client_negotiation.cc:769] Sending connection context
0928 16:49:29.371883 (+ 296us) client_negotiation.cc:240] Negotiation successful
0928 16:49:29.372447 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":353,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.3@KRBTEST.COM" created.
I20260928 16:49:29.375846 9964 heartbeater.cc:347] Connected to a master server at 127.9.93.62:42789
I20260928 16:49:29.376240 9964 heartbeater.cc:464] Registering TS with master...
I20260928 16:49:29.377156 9964 heartbeater.cc:511] Master 127.9.93.62:42789 requested a full tablet report, sending...
I20260928 16:49:29.379182 9640 ts_manager.cc:194] Registered new tserver with Master: bd4e3db393974a93bbd64610622d81b5 (127.9.93.2:35005)
I20260928 16:49:29.380136 9640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.2@KRBTEST.COM'} at 127.9.93.2:56773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
I20260928 16:49:29.399811 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:42789
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:37579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:29.778656 9970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:29.779021 9970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:29.779143 9970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:29.787266 9970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:29.787473 9970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:29.787626 9970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:49:29.792306 9830 heartbeater.cc:503] Master 127.9.93.62:42789 was elected leader, sending a full tablet report...
I20260928 16:49:29.799604 9970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:37579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:42789
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:29.801214 9970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:29.802867 9970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:29.818572 9976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:29.818532 9978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:29.818482 9975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:29.819155 9970 server_base.cc:1034] running on GCE node
I20260928 16:49:29.820029 9970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:29.821259 9970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:29.822510 9970 hybrid_clock.cc:648] HybridClock initialized: now 1790614169822443 us; error 50 us; skew 500 ppm
I20260928 16:49:29.822935 9970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:29 dist-test-slave-wzcb krb5kdc[9595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614169, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:29.826810 9970 init.cc:377] Logged in from keytab as kudu/127.9.93.3@KRBTEST.COM (short username kudu)
I20260928 16:49:29.828847 9970 webserver.cc:460] Webserver started at http://127.9.93.3:37223/ using document root <none> and password file <none>
I20260928 16:49:29.829727 9970 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:29.829874 9970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:29.830240 9970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:29.834112 9970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "44d4febcda204e079d60c5414dd6182a"
format_stamp: "Formatted at 2026-09-28 16:49:29 on dist-test-slave-wzcb"
server_key: "9a760d753833c2cee9f70c99737359c2"
server_key_iv: "be96ccd612c4daa71fd688a79b27279a"
server_key_version: "encryptionkey@0"
I20260928 16:49:29.835147 9970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "44d4febcda204e079d60c5414dd6182a"
format_stamp: "Formatted at 2026-09-28 16:49:29 on dist-test-slave-wzcb"
server_key: "9a760d753833c2cee9f70c99737359c2"
server_key_iv: "be96ccd612c4daa71fd688a79b27279a"
server_key_version: "encryptionkey@0"
I20260928 16:49:29.842365 9970 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260928 16:49:29.847182 9985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:29.849112 9970 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:49:29.849347 9970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "44d4febcda204e079d60c5414dd6182a"
format_stamp: "Formatted at 2026-09-28 16:49:29 on dist-test-slave-wzcb"
server_key: "9a760d753833c2cee9f70c99737359c2"
server_key_iv: "be96ccd612c4daa71fd688a79b27279a"
server_key_version: "encryptionkey@0"
I20260928 16:49:29.849606 9970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:29.864431 9970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:29.869673 9970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:29.870009 9970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:29.871268 9970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:49:29.873204 9970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:49:29.873302 9970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:29.873435 9970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:49:29.873492 9970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:29.910658 9970 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:45383
I20260928 16:49:29.910682 10097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:45383 every 8 connection(s)
I20260928 16:49:29.912249 9970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:49:29.921005 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 9970
I20260928 16:49:29.921213 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:49:29.921665 9588 external_mini_cluster.cc:1428] Setting key b05c275f1219e8e4c3dd26b3595973e8
Sep 28 16:49:29 dist-test-slave-wzcb krb5kdc[9595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614169, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:49:29.936717 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:29.915332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:45307 (local address 127.9.93.62:42789)
0928 16:49:29.915881 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:29.915886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:29.917280 (+ 1394us) server_negotiation.cc:407] Connection header received
0928 16:49:29.919477 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:29.919484 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:29.919598 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:29.919717 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:29.921756 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:29.922604 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:29.924941 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:29.925444 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:29.925566 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:29.929705 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:29.929711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:29.929715 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:29.929762 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:29.932662 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:29.933674 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:29.933680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:29.933682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:29.933836 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:29.934463 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:29.934468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:29.934471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:29.934855 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:29.934988 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:29.936020 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0928 16:49:29.936259 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":266,"threads_started":1}
I20260928 16:49:29.937947 10101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:29.916057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42789 (local address 127.9.93.3:45307)
0928 16:49:29.917036 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:29.917103 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:29.919109 (+ 2006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:29.920045 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:29.920055 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:29.920712 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:29.921540 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:29.921565 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:29.922813 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:29.922817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:29.924763 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:29.924782 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:29.925735 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:29.925743 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:29.926075 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:29.926752 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:29.926780 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:29.929376 (+ 2596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:29.932887 (+ 3511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:29.932904 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:29.932924 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:29.933508 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:29.934015 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:29.934020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:29.934022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:29.934304 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:29.935038 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:29.935046 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:29.935553 (+ 507us) client_negotiation.cc:769] Sending connection context
0928 16:49:29.935849 (+ 296us) client_negotiation.cc:240] Negotiation successful
0928 16:49:29.936442 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":369,"threads_started":1}
I20260928 16:49:29.939568 10098 heartbeater.cc:347] Connected to a master server at 127.9.93.62:42789
I20260928 16:49:29.939954 10098 heartbeater.cc:464] Registering TS with master...
I20260928 16:49:29.940845 10098 heartbeater.cc:511] Master 127.9.93.62:42789 requested a full tablet report, sending...
I20260928 16:49:29.942543 9640 ts_manager.cc:194] Registered new tserver with Master: 44d4febcda204e079d60c5414dd6182a (127.9.93.3:45383)
I20260928 16:49:29.943536 9640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.3@KRBTEST.COM'} at 127.9.93.3:45307
I20260928 16:49:29.952011 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 16:49:29 dist-test-slave-wzcb krb5kdc[9595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614169, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 16:49:29 dist-test-slave-wzcb krb5kdc[9595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614169, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.93.1@KRBTEST.COM
I20260928 16:49:30.002537 10106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:29.984894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38108 (local address 127.9.93.1:45771)
0928 16:49:29.985621 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:29.985629 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:29.985660 (+ 31us) server_negotiation.cc:407] Connection header received
0928 16:49:29.985822 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:29.985830 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:29.986089 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:29.986244 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:29.987908 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:29.989005 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:29.990417 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:29.991217 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:29.991378 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:29.995847 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:29.995867 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:29.995886 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:29.995948 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:29.999216 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:30.000170 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:30.000179 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:30.000182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:30.000349 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:30.000800 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:30.000804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:30.000806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:30.001258 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:30.001515 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:30.001843 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 16:49:30.002126 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":371,"threads_started":1}
W20260928 16:49:30.004196 9745 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38108
Sep 28 16:49:30 dist-test-slave-wzcb krb5kdc[9595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 16:49:30 dist-test-slave-wzcb krb5kdc[9595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.1@KRBTEST.COM
I20260928 16:49:30.055493 10106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:30.038904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38120 (local address 127.9.93.1:45771)
0928 16:49:30.039171 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:30.039178 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:30.039544 (+ 366us) server_negotiation.cc:407] Connection header received
0928 16:49:30.039779 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:30.039784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:30.039859 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:30.039951 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:30.041563 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:30.042359 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:30.043884 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:30.044426 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:30.044553 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:30.049088 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:30.049096 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:30.049100 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:30.049149 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:30.052192 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:30.053132 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:30.053138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:30.053141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:30.053298 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:30.053851 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:30.053858 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:30.053862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:30.054300 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:30.054523 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:30.054930 (+ 407us) server_negotiation.cc:299] Negotiation successful
0928 16:49:30.055180 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:49:30.057477 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 9702
I20260928 16:49:30.072410 10106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:30.058579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38126 (local address 127.9.93.1:45771)
0928 16:49:30.058749 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:30.058755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:30.059204 (+ 449us) server_negotiation.cc:407] Connection header received
0928 16:49:30.059418 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:30.059423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:30.059541 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:30.059636 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:30.061000 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:30.061773 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:30.063326 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:30.063847 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:30.063962 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:30.066550 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:30.066557 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:30.066561 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:30.066607 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:30.069397 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:30.070259 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:30.070264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:30.070267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:30.070450 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:30.070821 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:30.070824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:30.070826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:30.071156 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:30.071426 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:30.071843 (+ 417us) server_negotiation.cc:299] Negotiation successful
0928 16:49:30.072104 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 16:49:30.073240 9825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:49:30.161212 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 9702
I20260928 16:49:30.176553 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 9836
Sep 28 16:49:30 dist-test-slave-wzcb krb5kdc[9595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.2@KRBTEST.COM
I20260928 16:49:30.195442 10113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:30.177674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50582 (local address 127.9.93.2:35005)
0928 16:49:30.178256 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:30.178265 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:30.178294 (+ 29us) server_negotiation.cc:407] Connection header received
0928 16:49:30.178411 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:30.178419 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:30.178676 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:30.178835 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:30.180473 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:30.181601 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:30.183121 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:30.183886 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:30.184010 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:30.188758 (+ 4748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:30.188780 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:30.188800 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:30.188866 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:30.192246 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:30.193067 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:30.193077 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:30.193093 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:30.193264 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:30.193715 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:30.193718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:30.193720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:30.194145 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:30.194405 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:30.194712 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 16:49:30.194965 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":298,"threads_started":1}
I20260928 16:49:30.197045 9959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:49:30.285936 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 9836
I20260928 16:49:30.300637 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 9970
Sep 28 16:49:30 dist-test-slave-wzcb krb5kdc[9595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.3@KRBTEST.COM
I20260928 16:49:30.319380 10115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:30.301763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36622 (local address 127.9.93.3:45383)
0928 16:49:30.302539 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:30.302548 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:30.302583 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:49:30.302659 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:30.302667 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:30.302943 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:30.303137 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:30.304619 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:30.305750 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:30.307247 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:30.308019 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:30.308160 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:30.312657 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:30.312677 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:30.312696 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:30.312747 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:30.315926 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:30.316750 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:30.316761 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:30.316763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:30.316939 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:30.317363 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:30.317369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:30.317372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:30.317927 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:30.318286 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:30.318658 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 16:49:30.318948 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":359,"threads_started":1}
I20260928 16:49:30.321028 10093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:49:30.409116 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 9970
I20260928 16:49:30.423604 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 9608
I20260928 16:49:30.425045 9671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:49:30.533411 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 9608
2026-09-28T16:49:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4153 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:49:30 dist-test-slave-wzcb krb5kdc[10120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:49:30 dist-test-slave-wzcb krb5kdc[10120](info): set up 2 sockets
Sep 28 16:49:30 dist-test-slave-wzcb krb5kdc[10120](info): commencing operation
krb5kdc: starting...
W20260928 16:49:31.549870 9588 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:49:31 dist-test-slave-wzcb krb5kdc[10120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:49:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:49:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:49:31.747900 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:43823
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:43823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:32.102638 10133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:32.103039 10133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:32.103200 10133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:32.111080 10133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:32.111266 10133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:32.111399 10133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:32.111503 10133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:32.111577 10133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:32.123060 10133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:43823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:43823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:32.124759 10133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:32.126365 10133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:32.139212 10138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:32.139372 10141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:32.139338 10139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:32.140563 10133 server_base.cc:1034] running on GCE node
I20260928 16:49:32.141507 10133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:32.143270 10133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:32.144556 10133 hybrid_clock.cc:648] HybridClock initialized: now 1790614172144495 us; error 55 us; skew 500 ppm
I20260928 16:49:32.144958 10133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:32 dist-test-slave-wzcb krb5kdc[10120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614172, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:32.154006 10133 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:49:32.155951 10133 webserver.cc:460] Webserver started at http://127.9.93.62:44725/ using document root <none> and password file <none>
I20260928 16:49:32.156787 10133 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:32.156927 10133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:32.157281 10133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:32.160866 10133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "1fa7c9791c1847e982100d1e90c274f7"
format_stamp: "Formatted at 2026-09-28 16:49:32 on dist-test-slave-wzcb"
server_key: "b6bf6b453c6c0e921b6b464bb1af9881"
server_key_iv: "3dfc83da0139d436831f66ae2588ff90"
server_key_version: "encryptionkey@0"
I20260928 16:49:32.161753 10133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "1fa7c9791c1847e982100d1e90c274f7"
format_stamp: "Formatted at 2026-09-28 16:49:32 on dist-test-slave-wzcb"
server_key: "b6bf6b453c6c0e921b6b464bb1af9881"
server_key_iv: "3dfc83da0139d436831f66ae2588ff90"
server_key_version: "encryptionkey@0"
I20260928 16:49:32.168185 10133 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 16:49:32.172675 10148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:32.174645 10133 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:49:32.174883 10133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "1fa7c9791c1847e982100d1e90c274f7"
format_stamp: "Formatted at 2026-09-28 16:49:32 on dist-test-slave-wzcb"
server_key: "b6bf6b453c6c0e921b6b464bb1af9881"
server_key_iv: "3dfc83da0139d436831f66ae2588ff90"
server_key_version: "encryptionkey@0"
I20260928 16:49:32.175177 10133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:32.205744 10133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:32.211262 10133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:32.211655 10133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:32.230933 10133 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:43823
I20260928 16:49:32.230985 10200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:43823 every 8 connection(s)
I20260928 16:49:32.232611 10133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:49:32.236922 10201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:32.238901 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 10133
I20260928 16:49:32.239144 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:49:32.240005 9588 external_mini_cluster.cc:1428] Setting key 9c95416f164624b831416c619b85b2ab
I20260928 16:49:32.250402 10201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fa7c9791c1847e982100d1e90c274f7: Bootstrap starting.
I20260928 16:49:32.254045 10201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fa7c9791c1847e982100d1e90c274f7: Neither blocks nor log segments found. Creating new log.
Sep 28 16:49:32 dist-test-slave-wzcb krb5kdc[10120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:49:32.255487 10201 log.cc:824] T 00000000000000000000000000000000 P 1fa7c9791c1847e982100d1e90c274f7: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:32.259814 10201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fa7c9791c1847e982100d1e90c274f7: No bootstrap required, opened a new log
I20260928 16:49:32.266505 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:32.242494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49028 (local address 127.9.93.62:43823)
0928 16:49:32.243519 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:32.243534 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:32.243590 (+ 56us) server_negotiation.cc:407] Connection header received
0928 16:49:32.245470 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:32.245507 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:32.246008 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:32.246439 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:32.248223 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:32.249648 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:32.251052 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:32.251864 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:32.252094 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:32.256637 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:32.256647 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:32.256670 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:32.256722 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:32.260910 (+ 4188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:32.261795 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:32.261809 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:32.261820 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:32.262085 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:32.262463 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:32.262469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:32.262476 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:32.263425 (+ 949us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:32.263763 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:32.264292 (+ 529us) server_negotiation.cc:299] Negotiation successful
0928 16:49:32.264850 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":358,"threads_started":1}
I20260928 16:49:32.270700 10201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fa7c9791c1847e982100d1e90c274f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa7c9791c1847e982100d1e90c274f7" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43823 } }
I20260928 16:49:32.271138 10201 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fa7c9791c1847e982100d1e90c274f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:32.271230 10201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fa7c9791c1847e982100d1e90c274f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa7c9791c1847e982100d1e90c274f7, State: Initialized, Role: FOLLOWER
I20260928 16:49:32.271936 10201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fa7c9791c1847e982100d1e90c274f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa7c9791c1847e982100d1e90c274f7" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43823 } }
I20260928 16:49:32.272202 10201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fa7c9791c1847e982100d1e90c274f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:49:32.272308 10201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fa7c9791c1847e982100d1e90c274f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:49:32.272440 10201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fa7c9791c1847e982100d1e90c274f7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:32.274847 10201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fa7c9791c1847e982100d1e90c274f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa7c9791c1847e982100d1e90c274f7" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43823 } }
I20260928 16:49:32.275502 10201 leader_election.cc:302] T 00000000000000000000000000000000 P 1fa7c9791c1847e982100d1e90c274f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fa7c9791c1847e982100d1e90c274f7; no voters:
I20260928 16:49:32.276103 10201 leader_election.cc:288] T 00000000000000000000000000000000 P 1fa7c9791c1847e982100d1e90c274f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:49:32.276243 10206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fa7c9791c1847e982100d1e90c274f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:32.276898 10206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fa7c9791c1847e982100d1e90c274f7 [term 1 LEADER]: Becoming Leader. State: Replica: 1fa7c9791c1847e982100d1e90c274f7, State: Running, Role: LEADER
I20260928 16:49:32.277522 10206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fa7c9791c1847e982100d1e90c274f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa7c9791c1847e982100d1e90c274f7" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43823 } }
I20260928 16:49:32.278321 10201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fa7c9791c1847e982100d1e90c274f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:32.280859 10207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fa7c9791c1847e982100d1e90c274f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fa7c9791c1847e982100d1e90c274f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa7c9791c1847e982100d1e90c274f7" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43823 } } }
I20260928 16:49:32.280833 10208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fa7c9791c1847e982100d1e90c274f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fa7c9791c1847e982100d1e90c274f7. Latest consensus state: current_term: 1 leader_uuid: "1fa7c9791c1847e982100d1e90c274f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa7c9791c1847e982100d1e90c274f7" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43823 } } }
I20260928 16:49:32.281325 10207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fa7c9791c1847e982100d1e90c274f7 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:32.281323 10208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fa7c9791c1847e982100d1e90c274f7 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:32.284126 10214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:49:32.295689 10214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:49:32.307207 10214 catalog_manager.cc:1347] Generated new cluster ID: 3670380efd5f4a2b80604ed44384b671
I20260928 16:49:32.307368 10214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:49:32.317137 10214 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:49:32.319330 10214 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:49:32.330379 10214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fa7c9791c1847e982100d1e90c274f7: Generated new TSK 0
I20260928 16:49:32.331837 10214 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
I20260928 16:49:32.387815 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:43823
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:32.748343 10227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:32.748703 10227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:32.748848 10227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:32.756501 10227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:32.756665 10227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:32.756832 10227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:49:32.767851 10227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:43823
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:32.769394 10227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:32.771000 10227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:32.786955 10233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:32.786934 10232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:32.787041 10235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:32.787870 10227 server_base.cc:1034] running on GCE node
I20260928 16:49:32.788616 10227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:32.789791 10227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:32.791038 10227 hybrid_clock.cc:648] HybridClock initialized: now 1790614172790954 us; error 71 us; skew 500 ppm
I20260928 16:49:32.791481 10227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:32 dist-test-slave-wzcb krb5kdc[10120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614172, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:32.795209 10227 init.cc:377] Logged in from keytab as kudu/127.9.93.1@KRBTEST.COM (short username kudu)
I20260928 16:49:32.797582 10227 webserver.cc:460] Webserver started at http://127.9.93.1:41659/ using document root <none> and password file <none>
I20260928 16:49:32.798460 10227 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:32.798604 10227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:32.798954 10227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:32.802524 10227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "9786cb5c7aff4e238a39c00e16136b14"
format_stamp: "Formatted at 2026-09-28 16:49:32 on dist-test-slave-wzcb"
server_key: "f6b73c56dd00c6e80a0f32e49e7ae3fa"
server_key_iv: "5de840f3db74b4b9f40e98602eb123cb"
server_key_version: "encryptionkey@0"
I20260928 16:49:32.803503 10227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "9786cb5c7aff4e238a39c00e16136b14"
format_stamp: "Formatted at 2026-09-28 16:49:32 on dist-test-slave-wzcb"
server_key: "f6b73c56dd00c6e80a0f32e49e7ae3fa"
server_key_iv: "5de840f3db74b4b9f40e98602eb123cb"
server_key_version: "encryptionkey@0"
I20260928 16:49:32.809564 10227 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:49:32.813767 10242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:32.815680 10227 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:49:32.815884 10227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "9786cb5c7aff4e238a39c00e16136b14"
format_stamp: "Formatted at 2026-09-28 16:49:32 on dist-test-slave-wzcb"
server_key: "f6b73c56dd00c6e80a0f32e49e7ae3fa"
server_key_iv: "5de840f3db74b4b9f40e98602eb123cb"
server_key_version: "encryptionkey@0"
I20260928 16:49:32.816123 10227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:32.830607 10227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:32.835968 10227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:32.836316 10227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:32.837540 10227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:49:32.839244 10227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:49:32.839370 10227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260928 16:49:32.839509 10227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:49:32.839567 10227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:32.875847 10227 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:35503
I20260928 16:49:32.875885 10354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:35503 every 8 connection(s)
I20260928 16:49:32.877377 10227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:49:32.879526 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 10227
I20260928 16:49:32.879794 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:49:32.880685 9588 external_mini_cluster.cc:1428] Setting key dc9d167cf72aecc2202518ceb450c9d0
Sep 28 16:49:32 dist-test-slave-wzcb krb5kdc[10120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614172, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.2@KRBTEST.COM" created.
I20260928 16:49:32.903540 10357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:32.880791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:56393 (local address 127.9.93.62:43823)
0928 16:49:32.881274 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:32.881279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:32.882686 (+ 1407us) server_negotiation.cc:407] Connection header received
0928 16:49:32.885128 (+ 2442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:32.885134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:32.885225 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:32.885373 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:32.887126 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:32.888210 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:32.890998 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:32.891514 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:32.891631 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:32.896816 (+ 5185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:32.896823 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:32.896826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:32.896872 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:32.899618 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:32.900641 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:32.900646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:32.900649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:32.900799 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:32.901369 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:32.901373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:32.901376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:32.901704 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:32.901846 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:32.902802 (+ 956us) server_negotiation.cc:299] Negotiation successful
0928 16:49:32.903113 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":241,"threads_started":1}
I20260928 16:49:32.904814 10358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:32.881615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43823 (local address 127.9.93.1:56393)
0928 16:49:32.882493 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:32.882546 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:32.884775 (+ 2229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:32.885588 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:32.885598 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:32.886226 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:32.886908 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:32.886928 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:32.888404 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:32.888447 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:32.890780 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:32.890812 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:32.891830 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:32.891838 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:32.892204 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:32.893237 (+ 1033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:32.893274 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:32.896475 (+ 3201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:32.899833 (+ 3358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:32.899843 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:32.899862 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:32.900479 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:32.900945 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:32.900950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:32.900952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:32.901240 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:32.901864 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:32.901874 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:32.902321 (+ 447us) client_negotiation.cc:769] Sending connection context
0928 16:49:32.902669 (+ 348us) client_negotiation.cc:240] Negotiation successful
0928 16:49:32.903402 (+ 733us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":375,"threads_started":1}
I20260928 16:49:32.906854 10355 heartbeater.cc:347] Connected to a master server at 127.9.93.62:43823
I20260928 16:49:32.907248 10355 heartbeater.cc:464] Registering TS with master...
I20260928 16:49:32.908157 10355 heartbeater.cc:511] Master 127.9.93.62:43823 requested a full tablet report, sending...
I20260928 16:49:32.911007 10165 ts_manager.cc:194] Registered new tserver with Master: 9786cb5c7aff4e238a39c00e16136b14 (127.9.93.1:35503)
I20260928 16:49:32.912917 10165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.1@KRBTEST.COM'} at 127.9.93.1:56393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
I20260928 16:49:32.927193 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:43823
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:33.301360 10361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:33.301689 10361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:33.301801 10361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:33.309604 10361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:33.309744 10361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:33.309887 10361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:49:33.320762 10361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:43823
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:33.322243 10361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:33.323804 10361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:33.338845 10367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:33.338982 10366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:33.338963 10369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:33.339777 10361 server_base.cc:1034] running on GCE node
I20260928 16:49:33.340615 10361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:33.341838 10361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:33.343088 10361 hybrid_clock.cc:648] HybridClock initialized: now 1790614173343011 us; error 65 us; skew 500 ppm
I20260928 16:49:33.343542 10361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:33 dist-test-slave-wzcb krb5kdc[10120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614173, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:33.347220 10361 init.cc:377] Logged in from keytab as kudu/127.9.93.2@KRBTEST.COM (short username kudu)
I20260928 16:49:33.348938 10361 webserver.cc:460] Webserver started at http://127.9.93.2:39775/ using document root <none> and password file <none>
I20260928 16:49:33.349721 10361 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:33.349848 10361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:33.350203 10361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:33.353724 10361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "fc6f2dd789f54fc6acdd4920cd772082"
format_stamp: "Formatted at 2026-09-28 16:49:33 on dist-test-slave-wzcb"
server_key: "902f4688338afa8f27ace7425951751e"
server_key_iv: "f9971367fc14eb3d98d031ac310025d1"
server_key_version: "encryptionkey@0"
I20260928 16:49:33.354664 10361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "fc6f2dd789f54fc6acdd4920cd772082"
format_stamp: "Formatted at 2026-09-28 16:49:33 on dist-test-slave-wzcb"
server_key: "902f4688338afa8f27ace7425951751e"
server_key_iv: "f9971367fc14eb3d98d031ac310025d1"
server_key_version: "encryptionkey@0"
I20260928 16:49:33.360813 10361 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 16:49:33.364966 10376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:33.366572 10361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260928 16:49:33.366765 10361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "fc6f2dd789f54fc6acdd4920cd772082"
format_stamp: "Formatted at 2026-09-28 16:49:33 on dist-test-slave-wzcb"
server_key: "902f4688338afa8f27ace7425951751e"
server_key_iv: "f9971367fc14eb3d98d031ac310025d1"
server_key_version: "encryptionkey@0"
I20260928 16:49:33.366990 10361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:33.389456 10361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:33.394398 10361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:33.394717 10361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:33.395948 10361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:49:33.397472 10361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:49:33.397563 10361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:33.397696 10361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:49:33.397753 10361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:33.430640 10361 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:43643
I20260928 16:49:33.430701 10488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:43643 every 8 connection(s)
I20260928 16:49:33.432179 10361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:49:33.438535 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 10361
I20260928 16:49:33.438758 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:49:33.439349 9588 external_mini_cluster.cc:1428] Setting key ba056ca219a0d0a50d86cd68737b5f34
Sep 28 16:49:33 dist-test-slave-wzcb krb5kdc[10120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614173, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:49:33.457214 10491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:33.435069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:58585 (local address 127.9.93.62:43823)
0928 16:49:33.435622 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:33.435626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:33.436773 (+ 1147us) server_negotiation.cc:407] Connection header received
0928 16:49:33.439139 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:33.439144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:33.439224 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:33.439374 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:33.441429 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:33.442288 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:33.444890 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:33.445389 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:33.445496 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:33.450192 (+ 4696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:33.450198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:33.450201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:33.450245 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:33.453059 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:33.454270 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:33.454276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:33.454278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:33.454493 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:33.455115 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:33.455119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:33.455122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:33.455470 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:33.455606 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:33.456678 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0928 16:49:33.456882 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":320,"threads_started":1}
WARNING: no policy specified for kudu/127.9.93.3@KRBTEST.COM; defaulting to no policy
I20260928 16:49:33.459062 10492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:33.435789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43823 (local address 127.9.93.2:58585)
0928 16:49:33.436597 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:33.436647 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:33.438762 (+ 2115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:33.439698 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:33.439711 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:33.440487 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:33.441207 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:33.441233 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:33.442474 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:33.442480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:33.444701 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:33.444720 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:33.445660 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:33.445668 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:33.445981 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:33.446797 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:33.446836 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:33.449859 (+ 3023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:33.453267 (+ 3408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:33.453279 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:33.453302 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:33.454071 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:33.454637 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:33.454642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:33.454646 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:33.454972 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:33.455608 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:33.455618 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:33.456230 (+ 612us) client_negotiation.cc:769] Sending connection context
0928 16:49:33.456607 (+ 377us) client_negotiation.cc:240] Negotiation successful
0928 16:49:33.457381 (+ 774us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":299,"threads_started":1}
I20260928 16:49:33.460845 10489 heartbeater.cc:347] Connected to a master server at 127.9.93.62:43823
I20260928 16:49:33.461266 10489 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.3@KRBTEST.COM" created.
I20260928 16:49:33.462267 10489 heartbeater.cc:511] Master 127.9.93.62:43823 requested a full tablet report, sending...
I20260928 16:49:33.464161 10165 ts_manager.cc:194] Registered new tserver with Master: fc6f2dd789f54fc6acdd4920cd772082 (127.9.93.2:43643)
I20260928 16:49:33.465020 10165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.2@KRBTEST.COM'} at 127.9.93.2:58585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
I20260928 16:49:33.487111 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:43823
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:33.841853 10495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:33.842221 10495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:33.842379 10495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:33.850337 10495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:33.850505 10495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:33.850683 10495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:49:33.861651 10495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:43823
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:33.863288 10495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:33.864944 10495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:33.879742 10501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:33.879823 10500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:33.880506 10503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:33.881052 10495 server_base.cc:1034] running on GCE node
I20260928 16:49:33.881778 10495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:33.882913 10495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:33.884160 10495 hybrid_clock.cc:648] HybridClock initialized: now 1790614173884091 us; error 55 us; skew 500 ppm
I20260928 16:49:33.884553 10495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:33 dist-test-slave-wzcb krb5kdc[10120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614173, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:33.888226 10495 init.cc:377] Logged in from keytab as kudu/127.9.93.3@KRBTEST.COM (short username kudu)
I20260928 16:49:33.890147 10495 webserver.cc:460] Webserver started at http://127.9.93.3:45035/ using document root <none> and password file <none>
I20260928 16:49:33.890988 10495 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:33.891132 10495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:33.891580 10495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:33.895143 10495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "2d3b57ff809f4994881ead71d7894c58"
format_stamp: "Formatted at 2026-09-28 16:49:33 on dist-test-slave-wzcb"
server_key: "cb03abdc933bad089b820c5e603ae711"
server_key_iv: "c40b83fe5148bcdb20482a53ee2e6971"
server_key_version: "encryptionkey@0"
I20260928 16:49:33.896162 10495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "2d3b57ff809f4994881ead71d7894c58"
format_stamp: "Formatted at 2026-09-28 16:49:33 on dist-test-slave-wzcb"
server_key: "cb03abdc933bad089b820c5e603ae711"
server_key_iv: "c40b83fe5148bcdb20482a53ee2e6971"
server_key_version: "encryptionkey@0"
I20260928 16:49:33.902510 10495 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 16:49:33.906901 10510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:33.908697 10495 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260928 16:49:33.908908 10495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "2d3b57ff809f4994881ead71d7894c58"
format_stamp: "Formatted at 2026-09-28 16:49:33 on dist-test-slave-wzcb"
server_key: "cb03abdc933bad089b820c5e603ae711"
server_key_iv: "c40b83fe5148bcdb20482a53ee2e6971"
server_key_version: "encryptionkey@0"
I20260928 16:49:33.909150 10495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:33.916954 10355 heartbeater.cc:503] Master 127.9.93.62:43823 was elected leader, sending a full tablet report...
I20260928 16:49:33.934365 10495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:33.939394 10495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:33.939693 10495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:33.940800 10495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:49:33.942333 10495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:49:33.942420 10495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:33.942526 10495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:49:33.942572 10495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:33.977797 10495 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:42273
I20260928 16:49:33.977842 10623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:42273 every 8 connection(s)
I20260928 16:49:33.979378 10495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:49:33.987149 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 10495
I20260928 16:49:33.987378 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:49:33.987828 9588 external_mini_cluster.cc:1428] Setting key e12981f6b9118722b1a826744a10cd3b
Sep 28 16:49:33 dist-test-slave-wzcb krb5kdc[10120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614173, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:49:34.001696 10626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:33.982341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:42993 (local address 127.9.93.62:43823)
0928 16:49:33.982807 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:33.982812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:33.984042 (+ 1230us) server_negotiation.cc:407] Connection header received
0928 16:49:33.986131 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:33.986137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:33.986231 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:33.986335 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:33.988132 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:33.989200 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:33.991334 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:33.991804 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:33.991908 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:33.995634 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:33.995639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:33.995643 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:33.995681 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:33.998395 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:33.999342 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:33.999349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:33.999353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:33.999500 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:33.999909 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:33.999912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:33.999915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:34.000206 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:34.000328 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:34.001072 (+ 744us) server_negotiation.cc:299] Negotiation successful
0928 16:49:34.001295 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":247,"threads_started":1}
I20260928 16:49:34.002936 10627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:33.983025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43823 (local address 127.9.93.3:42993)
0928 16:49:33.983866 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:33.983923 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:33.985833 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:33.986606 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:33.986615 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:33.987208 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:33.987950 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:33.987976 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:33.989408 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:33.989414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:33.991154 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:33.991173 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:33.992030 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:33.992036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:33.992265 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:33.992909 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:33.992934 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:33.995378 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:33.998558 (+ 3180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:33.998567 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:33.998584 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:33.999177 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:33.999603 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:33.999607 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:33.999609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:33.999820 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:34.000307 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:34.000313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:34.000694 (+ 381us) client_negotiation.cc:769] Sending connection context
0928 16:49:34.000944 (+ 250us) client_negotiation.cc:240] Negotiation successful
0928 16:49:34.001446 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":327,"threads_started":1}
I20260928 16:49:34.004483 10624 heartbeater.cc:347] Connected to a master server at 127.9.93.62:43823
I20260928 16:49:34.004843 10624 heartbeater.cc:464] Registering TS with master...
I20260928 16:49:34.005730 10624 heartbeater.cc:511] Master 127.9.93.62:43823 requested a full tablet report, sending...
I20260928 16:49:34.007416 10165 ts_manager.cc:194] Registered new tserver with Master: 2d3b57ff809f4994881ead71d7894c58 (127.9.93.3:42273)
I20260928 16:49:34.008288 10165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.3@KRBTEST.COM'} at 127.9.93.3:42993
I20260928 16:49:34.016574 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:49:34.037817 10626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:34.023852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44520 (local address 127.9.93.62:43823)
0928 16:49:34.024032 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:34.024037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:34.024600 (+ 563us) server_negotiation.cc:407] Connection header received
0928 16:49:34.024788 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:34.024793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:34.024869 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:34.024951 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:34.026590 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.027462 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.028842 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.029281 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.029384 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:34.032066 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:34.032072 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:34.032076 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:34.032125 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:34.034910 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:34.035754 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:34.035759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:34.035762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:34.035902 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:34.036368 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:34.036372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:34.036374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:34.036736 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:34.036926 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:34.037330 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 16:49:34.037533 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:49:34.045835 10165 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:49:34.053529 10165 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:49:34.094686 10640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:34.068748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45372 (local address 127.9.93.3:42273)
0928 16:49:34.084395 (+ 15647us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:34.084405 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:34.084444 (+ 39us) server_negotiation.cc:407] Connection header received
0928 16:49:34.084560 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:34.084594 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:34.084919 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:34.085225 (+ 306us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:49:34.086906 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.088772 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.091216 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.092871 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.093017 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:34.093319 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:34.094073 (+ 754us) server_negotiation.cc:299] Negotiation successful
0928 16:49:34.094322 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15454,"thread_start_us":15178,"threads_started":1}
I20260928 16:49:34.094749 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:34.067921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:43643 (local address 127.0.0.1:42480)
0928 16:49:34.082626 (+ 14705us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:34.082706 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:34.082958 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:34.085663 (+ 2705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:34.085668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:34.085695 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:49:34.086183 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:34.086202 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.089011 (+ 2809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.089017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:34.091243 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:34.091258 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.093469 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.093478 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:34.093731 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:34.093829 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 16:49:34.093976 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 16:49:34.094223 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14311,"mutex_wait_us":37,"thread_start_us":15390,"threads_started":1}
I20260928 16:49:34.094813 10639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:34.068172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42480 (local address 127.9.93.2:43643)
0928 16:49:34.084565 (+ 16393us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:34.084571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:34.084599 (+ 28us) server_negotiation.cc:407] Connection header received
0928 16:49:34.084712 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:34.084719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:34.085011 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:34.085362 (+ 351us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:49:34.087253 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.088890 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.091436 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.092936 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.093055 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:34.093289 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:34.094254 (+ 965us) server_negotiation.cc:299] Negotiation successful
0928 16:49:34.094467 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16165,"thread_start_us":16017,"threads_started":1}
I20260928 16:49:34.094851 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:34.084785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:35503 (local address 127.0.0.1:42424)
0928 16:49:34.086501 (+ 1716us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:34.086530 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:34.086667 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:34.087855 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:34.087859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:34.087885 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:49:34.088256 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:34.088266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.090050 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.090055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:34.092131 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:34.092147 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.093913 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.093930 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:34.094110 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:34.094144 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 16:49:34.094234 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 16:49:34.094386 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1562,"mutex_wait_us":1,"thread_start_us":2167,"threads_started":1}
I20260928 16:49:34.095695 10641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:34.084585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42424 (local address 127.9.93.1:35503)
0928 16:49:34.087286 (+ 2701us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:34.087293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:34.087343 (+ 50us) server_negotiation.cc:407] Connection header received
0928 16:49:34.087432 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:34.087438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:34.087659 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:34.087847 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:49:34.088424 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.089885 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.092329 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.093679 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.093809 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:34.094111 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:34.095120 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0928 16:49:34.095356 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2512,"thread_start_us":2348,"threads_started":1}
I20260928 16:49:34.096203 10637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:34.068121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:42273 (local address 127.0.0.1:45372)
0928 16:49:34.083029 (+ 14908us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:34.083060 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:34.083226 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:34.085612 (+ 2386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:34.085622 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:34.085673 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:49:34.086662 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:34.086678 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.088936 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.088941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:34.091034 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:34.091048 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.093513 (+ 2465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.093520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:34.093712 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:34.093829 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 16:49:34.095622 (+ 1793us) client_negotiation.cc:240] Negotiation successful
0928 16:49:34.095775 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14742,"mutex_wait_us":498,"thread_start_us":15090,"threads_started":1}
I20260928 16:49:34.099720 10559 tablet_service.cc:1467] Processing CreateTablet for tablet 593618dd38f94cb4b6444004e9217398 (DEFAULT_TABLE table=test-table [id=8478690929184e97bda40d802417fbec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:49:34.099720 10424 tablet_service.cc:1467] Processing CreateTablet for tablet 593618dd38f94cb4b6444004e9217398 (DEFAULT_TABLE table=test-table [id=8478690929184e97bda40d802417fbec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:49:34.099720 10290 tablet_service.cc:1467] Processing CreateTablet for tablet 593618dd38f94cb4b6444004e9217398 (DEFAULT_TABLE table=test-table [id=8478690929184e97bda40d802417fbec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:49:34.101588 10559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 593618dd38f94cb4b6444004e9217398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:34.101684 10290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 593618dd38f94cb4b6444004e9217398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:34.101683 10424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 593618dd38f94cb4b6444004e9217398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:34.115670 10643 tablet_bootstrap.cc:492] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58: Bootstrap starting.
I20260928 16:49:34.118410 10644 tablet_bootstrap.cc:492] T 593618dd38f94cb4b6444004e9217398 P fc6f2dd789f54fc6acdd4920cd772082: Bootstrap starting.
I20260928 16:49:34.120209 10645 tablet_bootstrap.cc:492] T 593618dd38f94cb4b6444004e9217398 P 9786cb5c7aff4e238a39c00e16136b14: Bootstrap starting.
I20260928 16:49:34.122166 10643 tablet_bootstrap.cc:654] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:34.122740 10644 tablet_bootstrap.cc:654] T 593618dd38f94cb4b6444004e9217398 P fc6f2dd789f54fc6acdd4920cd772082: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:34.123893 10643 log.cc:824] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:34.123932 10644 log.cc:824] T 593618dd38f94cb4b6444004e9217398 P fc6f2dd789f54fc6acdd4920cd772082: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:34.124540 10645 tablet_bootstrap.cc:654] T 593618dd38f94cb4b6444004e9217398 P 9786cb5c7aff4e238a39c00e16136b14: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:34.126015 10645 log.cc:824] T 593618dd38f94cb4b6444004e9217398 P 9786cb5c7aff4e238a39c00e16136b14: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:34.127758 10644 tablet_bootstrap.cc:492] T 593618dd38f94cb4b6444004e9217398 P fc6f2dd789f54fc6acdd4920cd772082: No bootstrap required, opened a new log
I20260928 16:49:34.128082 10644 ts_tablet_manager.cc:1397] T 593618dd38f94cb4b6444004e9217398 P fc6f2dd789f54fc6acdd4920cd772082: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260928 16:49:34.128428 10643 tablet_bootstrap.cc:492] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58: No bootstrap required, opened a new log
I20260928 16:49:34.128831 10643 ts_tablet_manager.cc:1397] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.008s
I20260928 16:49:34.129531 10645 tablet_bootstrap.cc:492] T 593618dd38f94cb4b6444004e9217398 P 9786cb5c7aff4e238a39c00e16136b14: No bootstrap required, opened a new log
I20260928 16:49:34.129825 10645 ts_tablet_manager.cc:1397] T 593618dd38f94cb4b6444004e9217398 P 9786cb5c7aff4e238a39c00e16136b14: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 16:49:34.136998 10644 raft_consensus.cc:348] T 593618dd38f94cb4b6444004e9217398 P fc6f2dd789f54fc6acdd4920cd772082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6f2dd789f54fc6acdd4920cd772082" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43643 } } peers { permanent_uuid: "2d3b57ff809f4994881ead71d7894c58" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42273 } } peers { permanent_uuid: "9786cb5c7aff4e238a39c00e16136b14" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 35503 } }
I20260928 16:49:34.137473 10644 raft_consensus.cc:374] T 593618dd38f94cb4b6444004e9217398 P fc6f2dd789f54fc6acdd4920cd772082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:34.137624 10644 raft_consensus.cc:729] T 593618dd38f94cb4b6444004e9217398 P fc6f2dd789f54fc6acdd4920cd772082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc6f2dd789f54fc6acdd4920cd772082, State: Initialized, Role: FOLLOWER
I20260928 16:49:34.138420 10644 consensus_queue.cc:260] T 593618dd38f94cb4b6444004e9217398 P fc6f2dd789f54fc6acdd4920cd772082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6f2dd789f54fc6acdd4920cd772082" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43643 } } peers { permanent_uuid: "2d3b57ff809f4994881ead71d7894c58" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42273 } } peers { permanent_uuid: "9786cb5c7aff4e238a39c00e16136b14" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 35503 } }
I20260928 16:49:34.139698 10489 heartbeater.cc:503] Master 127.9.93.62:43823 was elected leader, sending a full tablet report...
I20260928 16:49:34.141029 10645 raft_consensus.cc:348] T 593618dd38f94cb4b6444004e9217398 P 9786cb5c7aff4e238a39c00e16136b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6f2dd789f54fc6acdd4920cd772082" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43643 } } peers { permanent_uuid: "2d3b57ff809f4994881ead71d7894c58" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42273 } } peers { permanent_uuid: "9786cb5c7aff4e238a39c00e16136b14" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 35503 } }
I20260928 16:49:34.141031 10643 raft_consensus.cc:348] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6f2dd789f54fc6acdd4920cd772082" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43643 } } peers { permanent_uuid: "2d3b57ff809f4994881ead71d7894c58" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42273 } } peers { permanent_uuid: "9786cb5c7aff4e238a39c00e16136b14" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 35503 } }
I20260928 16:49:34.141542 10643 raft_consensus.cc:374] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:34.141636 10645 raft_consensus.cc:374] T 593618dd38f94cb4b6444004e9217398 P 9786cb5c7aff4e238a39c00e16136b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:34.141711 10643 raft_consensus.cc:729] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d3b57ff809f4994881ead71d7894c58, State: Initialized, Role: FOLLOWER
I20260928 16:49:34.141820 10645 raft_consensus.cc:729] T 593618dd38f94cb4b6444004e9217398 P 9786cb5c7aff4e238a39c00e16136b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9786cb5c7aff4e238a39c00e16136b14, State: Initialized, Role: FOLLOWER
I20260928 16:49:34.142005 10644 ts_tablet_manager.cc:1428] T 593618dd38f94cb4b6444004e9217398 P fc6f2dd789f54fc6acdd4920cd772082: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260928 16:49:34.142557 10645 consensus_queue.cc:260] T 593618dd38f94cb4b6444004e9217398 P 9786cb5c7aff4e238a39c00e16136b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6f2dd789f54fc6acdd4920cd772082" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43643 } } peers { permanent_uuid: "2d3b57ff809f4994881ead71d7894c58" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42273 } } peers { permanent_uuid: "9786cb5c7aff4e238a39c00e16136b14" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 35503 } }
I20260928 16:49:34.142557 10643 consensus_queue.cc:260] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6f2dd789f54fc6acdd4920cd772082" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43643 } } peers { permanent_uuid: "2d3b57ff809f4994881ead71d7894c58" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42273 } } peers { permanent_uuid: "9786cb5c7aff4e238a39c00e16136b14" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 35503 } }
I20260928 16:49:34.144676 10643 ts_tablet_manager.cc:1428] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58: Time spent starting tablet: real 0.016s user 0.013s sys 0.002s
I20260928 16:49:34.144992 10645 ts_tablet_manager.cc:1428] T 593618dd38f94cb4b6444004e9217398 P 9786cb5c7aff4e238a39c00e16136b14: Time spent starting tablet: real 0.015s user 0.003s sys 0.011s
I20260928 16:49:34.145313 10624 heartbeater.cc:503] Master 127.9.93.62:43823 was elected leader, sending a full tablet report...
I20260928 16:49:34.170250 10650 raft_consensus.cc:482] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:49:34.170605 10650 raft_consensus.cc:504] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6f2dd789f54fc6acdd4920cd772082" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43643 } } peers { permanent_uuid: "2d3b57ff809f4994881ead71d7894c58" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42273 } } peers { permanent_uuid: "9786cb5c7aff4e238a39c00e16136b14" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 35503 } }
I20260928 16:49:34.173488 10650 leader_election.cc:288] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc6f2dd789f54fc6acdd4920cd772082 (127.9.93.2:43643), 9786cb5c7aff4e238a39c00e16136b14 (127.9.93.1:35503)
I20260928 16:49:34.181849 10641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:34.174279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:50027 (local address 127.9.93.1:35503)
0928 16:49:34.174536 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:34.174543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:34.175002 (+ 459us) server_negotiation.cc:407] Connection header received
0928 16:49:34.175147 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:34.175152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:34.175213 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:34.175284 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:49:34.175844 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.176833 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.179265 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.180678 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.180803 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:34.180901 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:34.181365 (+ 464us) server_negotiation.cc:299] Negotiation successful
0928 16:49:34.181528 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:49:34.181835 10639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:34.173950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:33019 (local address 127.9.93.2:43643)
0928 16:49:34.174233 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:34.174240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:34.174610 (+ 370us) server_negotiation.cc:407] Connection header received
0928 16:49:34.174726 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:34.174733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:34.174829 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:34.174914 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:49:34.175440 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.176464 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.179117 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.180585 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.180768 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:34.180886 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:34.181365 (+ 479us) server_negotiation.cc:299] Negotiation successful
0928 16:49:34.181541 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 16:49:34.181907 10627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:34.173975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:43643 (local address 127.9.93.3:33019)
0928 16:49:34.174368 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:34.174392 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:34.174564 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:34.174973 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:34.174978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:34.175070 (+ 92us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:49:34.175268 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:34.175280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.176627 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.176631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:34.178895 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:34.178917 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.180904 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.180912 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:34.181186 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:34.181212 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:49:34.181296 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 16:49:34.181522 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":148,"spinlock_wait_cycles":29056}
I20260928 16:49:34.181964 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:34.174025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:35503 (local address 127.9.93.3:50027)
0928 16:49:34.174891 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:34.174908 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:34.175025 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:34.175354 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:34.175359 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:34.175384 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:49:34.175676 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:34.175685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.176978 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.176986 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:34.179096 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:34.179118 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.180903 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.180912 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:34.181149 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:34.181190 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 16:49:34.181301 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 16:49:34.181522 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"mutex_wait_us":46,"thread_start_us":477,"threads_started":1}
I20260928 16:49:34.183441 10310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "593618dd38f94cb4b6444004e9217398" candidate_uuid: "2d3b57ff809f4994881ead71d7894c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9786cb5c7aff4e238a39c00e16136b14" is_pre_election: true
I20260928 16:49:34.184355 10310 raft_consensus.cc:2454] T 593618dd38f94cb4b6444004e9217398 P 9786cb5c7aff4e238a39c00e16136b14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d3b57ff809f4994881ead71d7894c58 in term 0.
W20260928 16:49:34.185253 10490 tablet.cc:2369] T 593618dd38f94cb4b6444004e9217398 P fc6f2dd789f54fc6acdd4920cd772082: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:49:34.185415 10513 leader_election.cc:302] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3b57ff809f4994881ead71d7894c58, 9786cb5c7aff4e238a39c00e16136b14; no voters:
I20260928 16:49:34.185320 10444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "593618dd38f94cb4b6444004e9217398" candidate_uuid: "2d3b57ff809f4994881ead71d7894c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc6f2dd789f54fc6acdd4920cd772082" is_pre_election: true
I20260928 16:49:34.185981 10650 raft_consensus.cc:2781] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:49:34.186012 10444 raft_consensus.cc:2454] T 593618dd38f94cb4b6444004e9217398 P fc6f2dd789f54fc6acdd4920cd772082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d3b57ff809f4994881ead71d7894c58 in term 0.
I20260928 16:49:34.186173 10650 raft_consensus.cc:482] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:49:34.186293 10650 raft_consensus.cc:3037] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:34.189018 10650 raft_consensus.cc:504] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6f2dd789f54fc6acdd4920cd772082" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43643 } } peers { permanent_uuid: "2d3b57ff809f4994881ead71d7894c58" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42273 } } peers { permanent_uuid: "9786cb5c7aff4e238a39c00e16136b14" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 35503 } }
I20260928 16:49:34.189765 10650 leader_election.cc:288] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58 [CANDIDATE]: Term 1 election: Requested vote from peers fc6f2dd789f54fc6acdd4920cd772082 (127.9.93.2:43643), 9786cb5c7aff4e238a39c00e16136b14 (127.9.93.1:35503)
I20260928 16:49:34.190436 10310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "593618dd38f94cb4b6444004e9217398" candidate_uuid: "2d3b57ff809f4994881ead71d7894c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9786cb5c7aff4e238a39c00e16136b14"
I20260928 16:49:34.190434 10444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "593618dd38f94cb4b6444004e9217398" candidate_uuid: "2d3b57ff809f4994881ead71d7894c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc6f2dd789f54fc6acdd4920cd772082"
I20260928 16:49:34.190726 10444 raft_consensus.cc:3037] T 593618dd38f94cb4b6444004e9217398 P fc6f2dd789f54fc6acdd4920cd772082 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:34.190709 10310 raft_consensus.cc:3037] T 593618dd38f94cb4b6444004e9217398 P 9786cb5c7aff4e238a39c00e16136b14 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:34.194387 10310 raft_consensus.cc:2454] T 593618dd38f94cb4b6444004e9217398 P 9786cb5c7aff4e238a39c00e16136b14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d3b57ff809f4994881ead71d7894c58 in term 1.
I20260928 16:49:34.194387 10444 raft_consensus.cc:2454] T 593618dd38f94cb4b6444004e9217398 P fc6f2dd789f54fc6acdd4920cd772082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d3b57ff809f4994881ead71d7894c58 in term 1.
I20260928 16:49:34.194993 10513 leader_election.cc:302] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3b57ff809f4994881ead71d7894c58, 9786cb5c7aff4e238a39c00e16136b14; no voters:
I20260928 16:49:34.195443 10650 raft_consensus.cc:2781] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:34.195968 10650 raft_consensus.cc:686] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58 [term 1 LEADER]: Becoming Leader. State: Replica: 2d3b57ff809f4994881ead71d7894c58, State: Running, Role: LEADER
I20260928 16:49:34.196547 10650 consensus_queue.cc:237] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6f2dd789f54fc6acdd4920cd772082" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43643 } } peers { permanent_uuid: "2d3b57ff809f4994881ead71d7894c58" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42273 } } peers { permanent_uuid: "9786cb5c7aff4e238a39c00e16136b14" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 35503 } }
I20260928 16:49:34.202344 10165 catalog_manager.cc:5495] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d3b57ff809f4994881ead71d7894c58 (127.9.93.3). New cstate: current_term: 1 leader_uuid: "2d3b57ff809f4994881ead71d7894c58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6f2dd789f54fc6acdd4920cd772082" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d3b57ff809f4994881ead71d7894c58" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9786cb5c7aff4e238a39c00e16136b14" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 35503 } health_report { overall_health: UNKNOWN } } }
W20260928 16:49:34.231577 10625 tablet.cc:2369] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 16:49:34 dist-test-slave-wzcb krb5kdc[10120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.1@KRBTEST.COM
I20260928 16:49:34.247381 10641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:34.228249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42430 (local address 127.9.93.1:35503)
0928 16:49:34.228521 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:34.228529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:34.229020 (+ 491us) server_negotiation.cc:407] Connection header received
0928 16:49:34.229196 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:34.229201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:34.229289 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:34.229379 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:34.231622 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.232842 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.234852 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.235427 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.235559 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:34.240813 (+ 5254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:34.240833 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:34.240852 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:34.240901 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:34.244110 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:34.244968 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:34.244978 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:34.244981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:34.245154 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:34.245578 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:34.245583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:34.245585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:34.246023 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:34.246335 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:34.246697 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 16:49:34.246951 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
Sep 28 16:49:34 dist-test-slave-wzcb krb5kdc[10120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614174, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 16:49:34 dist-test-slave-wzcb krb5kdc[10120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614174, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.93.1@KRBTEST.COM
I20260928 16:49:34.299186 10641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:34.283640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42436 (local address 127.9.93.1:35503)
0928 16:49:34.283871 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:34.283880 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:34.284343 (+ 463us) server_negotiation.cc:407] Connection header received
0928 16:49:34.284569 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:34.284573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:34.284653 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:34.284745 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:34.286266 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.287105 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.288530 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.288991 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.289116 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:34.293455 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:34.293462 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:34.293466 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:34.293521 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:34.296247 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:34.296994 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:34.297000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:34.297002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:34.297168 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:34.297609 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:34.297614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:34.297616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:34.297924 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:34.298188 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:34.298581 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 16:49:34.298845 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 16:49:34 dist-test-slave-wzcb krb5kdc[10120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 16:49:34 dist-test-slave-wzcb krb5kdc[10120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.1@KRBTEST.COM
I20260928 16:49:34.348873 10641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:34.333703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.9.93.1:35503)
0928 16:49:34.333962 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:34.333969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:34.334308 (+ 339us) server_negotiation.cc:407] Connection header received
0928 16:49:34.334524 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:34.334528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:34.334613 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:34.334704 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:34.336308 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.337050 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.338363 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.338845 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.338965 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:34.343157 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:34.343163 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:34.343166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:34.343212 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:34.345777 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:34.346574 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:34.346587 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:34.346590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:34.346742 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:34.347149 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:34.347154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:34.347156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:34.347564 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:34.347848 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:34.348175 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 16:49:34.348468 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 16:49:34.351473 10270 tablet_service.cc:2908] Rejecting scan request for tablet 593618dd38f94cb4b6444004e9217398: Uninitialized: clean time has not yet been initialized
I20260928 16:49:34.355532 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 10227
I20260928 16:49:34.369737 10641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:34.356597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42454 (local address 127.9.93.1:35503)
0928 16:49:34.356815 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:34.356820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:34.357117 (+ 297us) server_negotiation.cc:407] Connection header received
0928 16:49:34.357287 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:34.357291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:34.357366 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:34.357453 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:34.358826 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.359680 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.360969 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.361459 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.361589 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:34.364189 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:34.364195 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:34.364199 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:34.364244 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:34.366881 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:34.367804 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:34.367809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:34.367812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:34.367968 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:34.368284 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:34.368293 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:34.368296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:34.368673 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:34.368918 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:34.369199 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 16:49:34.369429 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:49:34.370558 10350 generic_service.cc:196] Checking for leaks (request via RPC)
W20260928 16:49:34.465329 10356 tablet.cc:2369] T 593618dd38f94cb4b6444004e9217398 P 9786cb5c7aff4e238a39c00e16136b14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:49:34.466148 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 10227
I20260928 16:49:34.482113 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 10361
Sep 28 16:49:34 dist-test-slave-wzcb krb5kdc[10120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.2@KRBTEST.COM
I20260928 16:49:34.499276 10639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:34.483024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42490 (local address 127.9.93.2:43643)
0928 16:49:34.483251 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:34.483257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:34.483279 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:49:34.483471 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:34.483475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:34.483603 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:34.483704 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:34.485098 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.485998 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.487556 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.488010 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.488147 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:34.492584 (+ 4437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:34.492608 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:34.492629 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:34.492683 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:34.496033 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:34.496970 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:34.496980 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:34.496983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:34.497153 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:34.497538 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:34.497542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:34.497544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:34.498032 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:34.498344 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:34.498740 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 16:49:34.498969 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 16:49:34.500113 10484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:49:34.595382 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 10361
I20260928 16:49:34.610719 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 10495
W20260928 16:49:34.610592 10512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.93.2:43643 (error 108)
I20260928 16:49:34.612916 10627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:34.612362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:43643 (local address 127.9.93.3:60355)
0928 16:49:34.612484 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:34.612703 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.2:43643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 16:49:34.613302 10512 consensus_peers.cc:487] T 593618dd38f94cb4b6444004e9217398 P 2d3b57ff809f4994881ead71d7894c58 -> Peer fc6f2dd789f54fc6acdd4920cd772082 (127.9.93.2:43643): Couldn't send request to peer fc6f2dd789f54fc6acdd4920cd772082. Status: Network error: Client connection negotiation failed: client connection to 127.9.93.2:43643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Sep 28 16:49:34 dist-test-slave-wzcb krb5kdc[10120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.3@KRBTEST.COM
I20260928 16:49:34.629027 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:34.611723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45384 (local address 127.9.93.3:42273)
0928 16:49:34.612312 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:34.612318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:34.612349 (+ 31us) server_negotiation.cc:407] Connection header received
0928 16:49:34.612776 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:34.612781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:34.612909 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:34.613018 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:34.614680 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.615619 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.617056 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:34.617543 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:34.617680 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:34.622400 (+ 4720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:34.622422 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:34.622441 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:34.622492 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:34.625573 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:34.626357 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:34.626365 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:34.626367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:34.626514 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:34.626979 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:34.626984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:34.626986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:34.627670 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:34.628009 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:34.628361 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 16:49:34.628645 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":292,"threads_started":1}
I20260928 16:49:34.629786 10619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:49:34.726591 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 10495
I20260928 16:49:34.741267 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 10133
I20260928 16:49:34.742549 10196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:49:34.855209 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 10133
2026-09-28T16:49:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4323 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:49:34 dist-test-slave-wzcb krb5kdc[10674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:49:34 dist-test-slave-wzcb krb5kdc[10674](info): set up 2 sockets
Sep 28 16:49:34 dist-test-slave-wzcb krb5kdc[10674](info): commencing operation
krb5kdc: starting...
W20260928 16:49:35.866384 9588 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:49:36 dist-test-slave-wzcb krb5kdc[10674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:49:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:49:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:49:36.077378 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:38095
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:38095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:36.439395 10687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:36.439787 10687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:36.439939 10687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:36.447875 10687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:36.448005 10687 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 16:49:36.448056 10687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:36.448149 10687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:36.448212 10687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:36.448284 10687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:36.459451 10687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:38095
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:38095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:36.461072 10687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:36.462702 10687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:36.476303 10692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:36.476249 10695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:36.476277 10693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:36.477314 10687 server_base.cc:1034] running on GCE node
I20260928 16:49:36.478289 10687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:36.480058 10687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:36.481328 10687 hybrid_clock.cc:648] HybridClock initialized: now 1790614176481245 us; error 53 us; skew 500 ppm
I20260928 16:49:36.481719 10687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:36 dist-test-slave-wzcb krb5kdc[10674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614176, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:36.485244 10687 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:49:36.486968 10687 webserver.cc:460] Webserver started at http://127.9.93.62:42831/ using document root <none> and password file <none>
I20260928 16:49:36.487861 10687 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:36.487985 10687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:36.488338 10687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:36.491863 10687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "7b45dbbb1946462a8537514ca7a9fc75"
format_stamp: "Formatted at 2026-09-28 16:49:36 on dist-test-slave-wzcb"
server_key: "9220e449217260109d38ba62cfe5aaed"
server_key_iv: "4e125d3a17f998a0a3cf0b34c18a68d9"
server_key_version: "encryptionkey@0"
I20260928 16:49:36.492815 10687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "7b45dbbb1946462a8537514ca7a9fc75"
format_stamp: "Formatted at 2026-09-28 16:49:36 on dist-test-slave-wzcb"
server_key: "9220e449217260109d38ba62cfe5aaed"
server_key_iv: "4e125d3a17f998a0a3cf0b34c18a68d9"
server_key_version: "encryptionkey@0"
I20260928 16:49:36.498919 10687 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 16:49:36.503428 10702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:36.505182 10687 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:49:36.505414 10687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "7b45dbbb1946462a8537514ca7a9fc75"
format_stamp: "Formatted at 2026-09-28 16:49:36 on dist-test-slave-wzcb"
server_key: "9220e449217260109d38ba62cfe5aaed"
server_key_iv: "4e125d3a17f998a0a3cf0b34c18a68d9"
server_key_version: "encryptionkey@0"
I20260928 16:49:36.505664 10687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:36.521242 10687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:36.525821 10687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:36.526139 10687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:36.549885 10687 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:38095
I20260928 16:49:36.549907 10763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:38095 every 8 connection(s)
I20260928 16:49:36.551592 10687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:49:36.556183 10764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:36.559049 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 10687
I20260928 16:49:36.559285 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:49:36.559985 9588 external_mini_cluster.cc:1428] Setting key b80ace630b584a3ab7129048e5cf80c7
I20260928 16:49:36.570690 10764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b45dbbb1946462a8537514ca7a9fc75: Bootstrap starting.
I20260928 16:49:36.574975 10764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b45dbbb1946462a8537514ca7a9fc75: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:36.576252 10764 log.cc:824] T 00000000000000000000000000000000 P 7b45dbbb1946462a8537514ca7a9fc75: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:49:36 dist-test-slave-wzcb krb5kdc[10674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:49:36.579732 10764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b45dbbb1946462a8537514ca7a9fc75: No bootstrap required, opened a new log
I20260928 16:49:36.587141 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:36.562487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46452 (local address 127.9.93.62:38095)
0928 16:49:36.563409 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:36.563429 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:36.563495 (+ 66us) server_negotiation.cc:407] Connection header received
0928 16:49:36.565901 (+ 2406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:36.565945 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:36.566622 (+ 677us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:36.567241 (+ 619us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:36.569125 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:36.571227 (+ 2102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:36.572796 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:36.573951 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:36.574175 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:36.578561 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:36.578570 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:36.578588 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:36.578641 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:36.582052 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:36.582951 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:36.582962 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:36.582970 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:36.583199 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:36.583673 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:36.583678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:36.583680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:36.584372 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:36.584749 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:36.585292 (+ 543us) server_negotiation.cc:299] Negotiation successful
0928 16:49:36.585837 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":375,"threads_started":1}
I20260928 16:49:36.588840 10764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b45dbbb1946462a8537514ca7a9fc75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b45dbbb1946462a8537514ca7a9fc75" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 38095 } }
I20260928 16:49:36.589422 10764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b45dbbb1946462a8537514ca7a9fc75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:36.589556 10764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b45dbbb1946462a8537514ca7a9fc75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b45dbbb1946462a8537514ca7a9fc75, State: Initialized, Role: FOLLOWER
I20260928 16:49:36.590329 10764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b45dbbb1946462a8537514ca7a9fc75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b45dbbb1946462a8537514ca7a9fc75" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 38095 } }
I20260928 16:49:36.590673 10764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b45dbbb1946462a8537514ca7a9fc75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:49:36.590827 10764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b45dbbb1946462a8537514ca7a9fc75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:49:36.591029 10764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b45dbbb1946462a8537514ca7a9fc75 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:36.593526 10764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b45dbbb1946462a8537514ca7a9fc75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b45dbbb1946462a8537514ca7a9fc75" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 38095 } }
I20260928 16:49:36.594270 10764 leader_election.cc:302] T 00000000000000000000000000000000 P 7b45dbbb1946462a8537514ca7a9fc75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b45dbbb1946462a8537514ca7a9fc75; no voters:
I20260928 16:49:36.594998 10764 leader_election.cc:288] T 00000000000000000000000000000000 P 7b45dbbb1946462a8537514ca7a9fc75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:49:36.595171 10769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b45dbbb1946462a8537514ca7a9fc75 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:36.596036 10769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b45dbbb1946462a8537514ca7a9fc75 [term 1 LEADER]: Becoming Leader. State: Replica: 7b45dbbb1946462a8537514ca7a9fc75, State: Running, Role: LEADER
I20260928 16:49:36.596670 10769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b45dbbb1946462a8537514ca7a9fc75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b45dbbb1946462a8537514ca7a9fc75" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 38095 } }
I20260928 16:49:36.597373 10764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b45dbbb1946462a8537514ca7a9fc75 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:36.599975 10770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b45dbbb1946462a8537514ca7a9fc75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b45dbbb1946462a8537514ca7a9fc75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b45dbbb1946462a8537514ca7a9fc75" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 38095 } } }
I20260928 16:49:36.600558 10770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b45dbbb1946462a8537514ca7a9fc75 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:36.600410 10771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b45dbbb1946462a8537514ca7a9fc75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b45dbbb1946462a8537514ca7a9fc75. Latest consensus state: current_term: 1 leader_uuid: "7b45dbbb1946462a8537514ca7a9fc75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b45dbbb1946462a8537514ca7a9fc75" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 38095 } } }
I20260928 16:49:36.600689 10771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b45dbbb1946462a8537514ca7a9fc75 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:36.601732 10776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:49:36.607790 10776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:49:36.619889 10776 catalog_manager.cc:1347] Generated new cluster ID: 6fd08f3fdab5476ea8bdc08c9cd81ef5
I20260928 16:49:36.620038 10776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:49:36.655221 10776 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:49:36.656816 10776 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:49:36.663568 10776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b45dbbb1946462a8537514ca7a9fc75: Generated new TSK 0
I20260928 16:49:36.664573 10776 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
I20260928 16:49:36.727522 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:38095
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:37.118810 10790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:37.119163 10790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:37.119333 10790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:37.127789 10790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:37.127944 10790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:37.128113 10790 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:49:37.128249 10790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:49:37.139667 10790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.93.62:38095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:37.141381 10790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:37.142961 10790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:37.158113 10795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:37.158198 10796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:37.158154 10798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:37.159122 10790 server_base.cc:1034] running on GCE node
I20260928 16:49:37.159976 10790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:37.161175 10790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:37.162454 10790 hybrid_clock.cc:648] HybridClock initialized: now 1790614177162350 us; error 73 us; skew 500 ppm
I20260928 16:49:37.162839 10790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:37 dist-test-slave-wzcb krb5kdc[10674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614177, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:37.172494 10790 init.cc:377] Logged in from keytab as kudu/127.9.93.1@KRBTEST.COM (short username kudu)
I20260928 16:49:37.174393 10790 webserver.cc:460] Webserver started at http://127.9.93.1:35711/ using document root <none> and password file <none>
I20260928 16:49:37.175371 10790 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:37.175539 10790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:37.175936 10790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:37.179541 10790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "0afab931743a4802b0dfbcc0e95680b1"
format_stamp: "Formatted at 2026-09-28 16:49:37 on dist-test-slave-wzcb"
server_key: "1482ab0a28fc0cf64064b1aba4ba5170"
server_key_iv: "6ce69f82eff4e79db69fd6f4bdc0f2e4"
server_key_version: "encryptionkey@0"
I20260928 16:49:37.180574 10790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "0afab931743a4802b0dfbcc0e95680b1"
format_stamp: "Formatted at 2026-09-28 16:49:37 on dist-test-slave-wzcb"
server_key: "1482ab0a28fc0cf64064b1aba4ba5170"
server_key_iv: "6ce69f82eff4e79db69fd6f4bdc0f2e4"
server_key_version: "encryptionkey@0"
I20260928 16:49:37.186913 10790 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 16:49:37.191570 10805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:37.193436 10790 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:49:37.193689 10790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "0afab931743a4802b0dfbcc0e95680b1"
format_stamp: "Formatted at 2026-09-28 16:49:37 on dist-test-slave-wzcb"
server_key: "1482ab0a28fc0cf64064b1aba4ba5170"
server_key_iv: "6ce69f82eff4e79db69fd6f4bdc0f2e4"
server_key_version: "encryptionkey@0"
I20260928 16:49:37.194096 10790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:37.209640 10790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:37.214704 10790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:37.215034 10790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:37.218560 10790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:49:37.218705 10790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:37.218864 10790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:49:37.218937 10790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:49:37 dist-test-slave-wzcb krb5kdc[10674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614177, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:49:37.246433 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:37.219461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:44217 (local address 127.9.93.62:38095)
0928 16:49:37.219970 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:37.219975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:37.221592 (+ 1617us) server_negotiation.cc:407] Connection header received
0928 16:49:37.224000 (+ 2408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:37.224006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:37.224108 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:37.224219 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:37.227046 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:37.227978 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:37.231188 (+ 3210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:37.231838 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:37.231989 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:37.238146 (+ 6157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:37.238156 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:37.238162 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:37.238227 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:37.242268 (+ 4041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:37.243392 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:37.243402 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:37.243406 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:37.243634 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:37.244375 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:37.244379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:37.244382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:37.244702 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:37.244849 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:37.245762 (+ 913us) server_negotiation.cc:299] Negotiation successful
0928 16:49:37.246041 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":280,"threads_started":1}
I20260928 16:49:37.247867 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:37.220342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:38095 (local address 127.9.93.1:44217)
0928 16:49:37.221386 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:37.221433 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:37.223632 (+ 2199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:37.224590 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:37.224610 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:37.225621 (+ 1011us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:37.226792 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:37.226824 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:37.228219 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:37.228227 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:37.230874 (+ 2647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:37.230910 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:37.232170 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:37.232179 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:37.232481 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:37.233426 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:37.233470 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:37.237689 (+ 4219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:37.242515 (+ 4826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:37.242528 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:37.242586 (+ 58us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:37.243201 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:37.243819 (+ 618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:37.243826 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:37.243829 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:37.244223 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:37.244872 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:37.244882 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:37.245329 (+ 447us) client_negotiation.cc:769] Sending connection context
0928 16:49:37.245615 (+ 286us) client_negotiation.cc:240] Negotiation successful
0928 16:49:37.246312 (+ 697us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":463,"threads_started":1}
I20260928 16:49:37.273046 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:37.255489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:49177 (local address 127.9.93.62:38095)
0928 16:49:37.255716 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:37.255723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:37.256974 (+ 1251us) server_negotiation.cc:407] Connection header received
0928 16:49:37.257206 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:37.257211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:37.257288 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:37.257379 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:37.258903 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:37.259760 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:37.262353 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:37.262871 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:37.263023 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:37.265880 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:37.265887 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:37.265891 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:37.265934 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:37.269359 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:37.270522 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:37.270530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:37.270534 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:37.270722 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:37.271269 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:37.271275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:37.271279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:37.271749 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:37.271911 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:37.272384 (+ 473us) server_negotiation.cc:299] Negotiation successful
0928 16:49:37.272657 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 16:49:37.273182 10887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:37.255378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:38095 (local address 127.9.93.1:49177)
0928 16:49:37.256793 (+ 1415us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:37.256820 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:37.257063 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:37.257439 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:37.257445 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:37.258185 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:37.258682 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:37.258704 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:37.259971 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:37.259977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:37.262095 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:37.262130 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:37.263146 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:37.263155 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:37.263413 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:37.264147 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:37.264181 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:37.265541 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:37.269607 (+ 4066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:37.269614 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:37.269622 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:37.270290 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:37.270856 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:37.270862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:37.270869 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:37.271084 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:37.271890 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:37.271895 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:37.272110 (+ 215us) client_negotiation.cc:769] Sending connection context
0928 16:49:37.272249 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 16:49:37.272691 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1192,"thread_start_us":911,"threads_started":1}
I20260928 16:49:37.277923 10790 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:44601
I20260928 16:49:37.277942 10925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:44601 every 8 connection(s)
I20260928 16:49:37.279428 10790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:49:37.281724 10926 heartbeater.cc:347] Connected to a master server at 127.9.93.62:38095
I20260928 16:49:37.281968 10926 heartbeater.cc:464] Registering TS with master...
I20260928 16:49:37.282702 10926 heartbeater.cc:511] Master 127.9.93.62:38095 requested a full tablet report, sending...
I20260928 16:49:37.283969 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 10790
I20260928 16:49:37.284185 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:49:37.284854 9588 external_mini_cluster.cc:1428] Setting key 3ea8812002d626dc6a4e9b818e907b5a
I20260928 16:49:37.285175 10719 ts_manager.cc:194] Registered new tserver with Master: 0afab931743a4802b0dfbcc0e95680b1 (127.9.93.1:44601)
I20260928 16:49:37.287453 10719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.1@KRBTEST.COM'} at 127.9.93.1:44217
WARNING: no policy specified for kudu/127.9.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
I20260928 16:49:37.330919 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:38095
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:37.734867 10930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:37.735252 10930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:37.735440 10930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:37.744077 10930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:37.744258 10930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:37.744400 10930 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:49:37.744520 10930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:49:37.756392 10930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.93.62:38095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:37.757992 10930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:37.759744 10930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:37.775736 10935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:37.775732 10936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:37.775905 10938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:37.776672 10930 server_base.cc:1034] running on GCE node
I20260928 16:49:37.777534 10930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:37.778787 10930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:37.780066 10930 hybrid_clock.cc:648] HybridClock initialized: now 1790614177779975 us; error 65 us; skew 500 ppm
I20260928 16:49:37.780467 10930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:37 dist-test-slave-wzcb krb5kdc[10674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614177, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:37.784317 10930 init.cc:377] Logged in from keytab as kudu/127.9.93.2@KRBTEST.COM (short username kudu)
I20260928 16:49:37.786190 10930 webserver.cc:460] Webserver started at http://127.9.93.2:40371/ using document root <none> and password file <none>
I20260928 16:49:37.787076 10930 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:37.787226 10930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:37.787657 10930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:37.791374 10930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "74e320993c594fb5b5aae9684584cc6e"
format_stamp: "Formatted at 2026-09-28 16:49:37 on dist-test-slave-wzcb"
server_key: "d48d2258df6ce7002c380dff67fce117"
server_key_iv: "b3341ac1c34fdbaabf99c14f9224048a"
server_key_version: "encryptionkey@0"
I20260928 16:49:37.792557 10930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "74e320993c594fb5b5aae9684584cc6e"
format_stamp: "Formatted at 2026-09-28 16:49:37 on dist-test-slave-wzcb"
server_key: "d48d2258df6ce7002c380dff67fce117"
server_key_iv: "b3341ac1c34fdbaabf99c14f9224048a"
server_key_version: "encryptionkey@0"
I20260928 16:49:37.799530 10930 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260928 16:49:37.804632 10945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:37.806559 10930 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:49:37.806802 10930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "74e320993c594fb5b5aae9684584cc6e"
format_stamp: "Formatted at 2026-09-28 16:49:37 on dist-test-slave-wzcb"
server_key: "d48d2258df6ce7002c380dff67fce117"
server_key_iv: "b3341ac1c34fdbaabf99c14f9224048a"
server_key_version: "encryptionkey@0"
I20260928 16:49:37.807085 10930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:37.841802 10930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:37.847190 10930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:37.847560 10930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:37.850910 10930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:49:37.851053 10930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:37.851212 10930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:49:37.851281 10930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:49:37 dist-test-slave-wzcb krb5kdc[10674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614177, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:49:37.881074 10954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:37.851985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:33929 (local address 127.9.93.62:38095)
0928 16:49:37.852486 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:37.852490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:37.854203 (+ 1713us) server_negotiation.cc:407] Connection header received
0928 16:49:37.857386 (+ 3183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:37.857394 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:37.857560 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:37.857698 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:37.860859 (+ 3161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:37.862206 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:37.865188 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:37.865854 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:37.866028 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:37.872751 (+ 6723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:37.872762 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:37.872768 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:37.872832 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:37.876762 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:37.877934 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:37.877939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:37.877942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:37.878112 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:37.878669 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:37.878674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:37.878676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:37.879006 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:37.879160 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:37.880335 (+ 1175us) server_negotiation.cc:299] Negotiation successful
0928 16:49:37.880648 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":263,"threads_started":1}
I20260928 16:49:37.883342 10955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:37.852824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:38095 (local address 127.9.93.2:33929)
0928 16:49:37.853957 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:37.854021 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:37.856883 (+ 2862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:37.858081 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:37.858094 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:37.859185 (+ 1091us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:37.860470 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:37.860501 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:37.862463 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:37.862476 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:37.864842 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:37.864865 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:37.866212 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:37.866224 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:37.866673 (+ 449us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:37.867789 (+ 1116us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:37.867840 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:37.872270 (+ 4430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:37.877102 (+ 4832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:37.877116 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:37.877145 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:37.877748 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:37.878247 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:37.878252 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:37.878254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:37.878523 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:37.879161 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:37.879171 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:37.879752 (+ 581us) client_negotiation.cc:769] Sending connection context
0928 16:49:37.880182 (+ 430us) client_negotiation.cc:240] Negotiation successful
0928 16:49:37.880994 (+ 812us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":510,"threads_started":1}
I20260928 16:49:37.908550 10930 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:33345
I20260928 16:49:37.908576 11066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:33345 every 8 connection(s)
I20260928 16:49:37.908826 10954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:37.891061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:60885 (local address 127.9.93.62:38095)
0928 16:49:37.891287 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:37.891294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:37.892080 (+ 786us) server_negotiation.cc:407] Connection header received
0928 16:49:37.892355 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:37.892362 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:37.892467 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:37.892600 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:37.894003 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:37.895218 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:37.897960 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:37.898515 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:37.898641 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:37.901579 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:37.901586 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:37.901590 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:37.901632 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:37.904951 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:37.906129 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:37.906137 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:37.906142 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:37.906312 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:37.906788 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:37.906793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:37.906795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:37.907408 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:37.907589 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:37.908172 (+ 583us) server_negotiation.cc:299] Negotiation successful
0928 16:49:37.908473 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 16:49:37.909181 11040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:37.891101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:38095 (local address 127.9.93.2:60885)
0928 16:49:37.891935 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:37.891965 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:37.892189 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:37.892631 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:37.892635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:37.893339 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:37.893783 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:37.893796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:37.895577 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:37.895584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:37.897697 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:37.897728 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:37.898793 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:37.898801 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:37.899025 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:37.899904 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:37.899941 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:37.901253 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:37.905145 (+ 3892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:37.905153 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:37.905158 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:37.905951 (+ 793us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:37.906442 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:37.906447 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:37.906451 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:37.906635 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:37.907660 (+ 1025us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:37.907665 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:37.907876 (+ 211us) client_negotiation.cc:769] Sending connection context
0928 16:49:37.908046 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 16:49:37.908639 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":491,"threads_started":1}
I20260928 16:49:37.910476 10930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:49:37.913012 11067 heartbeater.cc:347] Connected to a master server at 127.9.93.62:38095
I20260928 16:49:37.913286 11067 heartbeater.cc:464] Registering TS with master...
I20260928 16:49:37.914109 11067 heartbeater.cc:511] Master 127.9.93.62:38095 requested a full tablet report, sending...
I20260928 16:49:37.914510 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 10930
I20260928 16:49:37.914729 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:49:37.915282 9588 external_mini_cluster.cc:1428] Setting key fea70872f546cd2a061227d54dd6cb3d
I20260928 16:49:37.916014 10719 ts_manager.cc:194] Registered new tserver with Master: 74e320993c594fb5b5aae9684584cc6e (127.9.93.2:33345)
I20260928 16:49:37.917289 10719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.2@KRBTEST.COM'} at 127.9.93.2:33929
WARNING: no policy specified for kudu/127.9.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
I20260928 16:49:37.964260 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:38095
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:49:38.290912 10926 heartbeater.cc:503] Master 127.9.93.62:38095 was elected leader, sending a full tablet report...
W20260928 16:49:38.356129 11071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:38.356469 11071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:38.356621 11071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:38.365096 11071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:38.365240 11071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:38.365367 11071 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:49:38.365484 11071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:49:38.377278 11071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.93.62:38095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:38.378882 11071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:38.380582 11071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:38.396898 11077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:38.396893 11076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:38.398990 11079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:38.401081 11071 server_base.cc:1034] running on GCE node
I20260928 16:49:38.401959 11071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:38.403194 11071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:38.406646 11071 hybrid_clock.cc:648] HybridClock initialized: now 1790614178406502 us; error 93 us; skew 500 ppm
I20260928 16:49:38.407074 11071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:38 dist-test-slave-wzcb krb5kdc[10674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614178, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:38.411777 11071 init.cc:377] Logged in from keytab as kudu/127.9.93.3@KRBTEST.COM (short username kudu)
I20260928 16:49:38.414337 11071 webserver.cc:460] Webserver started at http://127.9.93.3:45823/ using document root <none> and password file <none>
I20260928 16:49:38.416375 11071 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:38.416640 11071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:38.417136 11071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:38.422346 11071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde"
format_stamp: "Formatted at 2026-09-28 16:49:38 on dist-test-slave-wzcb"
server_key: "05e4a0e6e2a3d81b60ae8d5b3921efeb"
server_key_iv: "d42fcb8eadb230ee1ede2337d6782304"
server_key_version: "encryptionkey@0"
I20260928 16:49:38.423871 11071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde"
format_stamp: "Formatted at 2026-09-28 16:49:38 on dist-test-slave-wzcb"
server_key: "05e4a0e6e2a3d81b60ae8d5b3921efeb"
server_key_iv: "d42fcb8eadb230ee1ede2337d6782304"
server_key_version: "encryptionkey@0"
I20260928 16:49:38.433015 11071 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20260928 16:49:38.438844 11086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:38.440924 11071 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 16:49:38.441187 11071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde"
format_stamp: "Formatted at 2026-09-28 16:49:38 on dist-test-slave-wzcb"
server_key: "05e4a0e6e2a3d81b60ae8d5b3921efeb"
server_key_iv: "d42fcb8eadb230ee1ede2337d6782304"
server_key_version: "encryptionkey@0"
I20260928 16:49:38.441470 11071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:38.458474 11071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:38.464109 11071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:38.464442 11071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:38.468032 11071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:49:38.468227 11071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 16:49:38.468413 11071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:49:38.468508 11071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:49:38 dist-test-slave-wzcb krb5kdc[10674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614178, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:49:38.519006 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:38.469602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:53915 (local address 127.9.93.62:38095)
0928 16:49:38.486802 (+ 17200us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:38.486809 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:38.486836 (+ 27us) server_negotiation.cc:407] Connection header received
0928 16:49:38.490629 (+ 3793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:38.490638 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:38.490765 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:38.490864 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:38.494669 (+ 3805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.496116 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.500018 (+ 3902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.500760 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.500900 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:38.507870 (+ 6970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:38.507880 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:38.507884 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:38.507939 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:38.512465 (+ 4526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:38.513932 (+ 1467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:38.513940 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:38.513945 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:38.514219 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:38.514926 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:38.514933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:38.514937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:38.515448 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:38.515622 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:38.518228 (+ 2606us) server_negotiation.cc:299] Negotiation successful
0928 16:49:38.518526 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17065,"thread_start_us":16939,"threads_started":1}
I20260928 16:49:38.520947 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:38.470269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:38095 (local address 127.9.93.3:53915)
0928 16:49:38.485899 (+ 15630us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:38.486002 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:38.490036 (+ 4034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:38.491771 (+ 1735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:38.491789 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:38.492968 (+ 1179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:38.494314 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:38.494354 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.496386 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.496393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:38.499661 (+ 3268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:38.499706 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.501387 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.501395 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:38.501788 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:38.502803 (+ 1015us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:38.502842 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:38.507421 (+ 4579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:38.512786 (+ 5365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:38.512805 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:38.512831 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:38.513712 (+ 881us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:38.514378 (+ 666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:38.514383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:38.514387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:38.514747 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:38.515626 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:38.515636 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:38.516347 (+ 711us) client_negotiation.cc:769] Sending connection context
0928 16:49:38.516774 (+ 427us) client_negotiation.cc:240] Negotiation successful
0928 16:49:38.517622 (+ 848us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15184,"thread_start_us":14863,"threads_started":1}
I20260928 16:49:38.554142 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:38.531590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:36791 (local address 127.9.93.62:38095)
0928 16:49:38.531808 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:38.531814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:38.532850 (+ 1036us) server_negotiation.cc:407] Connection header received
0928 16:49:38.533218 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:38.533224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:38.533338 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:38.533429 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:38.535498 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.536772 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.539525 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.540344 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.540505 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:38.545183 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:38.545193 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:38.545198 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:38.545287 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:38.549929 (+ 4642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:38.551159 (+ 1230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:38.551167 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:38.551172 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:38.551443 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:38.552096 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:38.552103 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:38.552107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:38.552604 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:38.552783 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:38.553360 (+ 577us) server_negotiation.cc:299] Negotiation successful
0928 16:49:38.553669 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 16:49:38.554242 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:38.531108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:38095 (local address 127.9.93.3:36791)
0928 16:49:38.532696 (+ 1588us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:38.532723 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:38.533013 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:38.533715 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:38.533721 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:38.534578 (+ 857us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:38.535191 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:38.535215 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.537020 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.537030 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:38.539161 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:38.539195 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.540875 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.540884 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:38.541149 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:38.542193 (+ 1044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:38.542228 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:38.544537 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:38.550178 (+ 5641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:38.550186 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:38.550191 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:38.550975 (+ 784us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:38.551611 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:38.551617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:38.551620 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:38.551834 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:38.552768 (+ 934us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:38.552774 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:38.553057 (+ 283us) client_negotiation.cc:769] Sending connection context
0928 16:49:38.553233 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 16:49:38.553691 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1377,"thread_start_us":1186,"threads_started":1}
I20260928 16:49:38.564827 11071 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:33469
I20260928 16:49:38.564842 11206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:33469 every 8 connection(s)
I20260928 16:49:38.566304 11071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:49:38.569391 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 11071
I20260928 16:49:38.569612 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:49:38.570199 9588 external_mini_cluster.cc:1428] Setting key 2fce8accc889f2314a84a771130bc5c1
I20260928 16:49:38.570962 11207 heartbeater.cc:347] Connected to a master server at 127.9.93.62:38095
I20260928 16:49:38.571278 11207 heartbeater.cc:464] Registering TS with master...
I20260928 16:49:38.572314 11207 heartbeater.cc:511] Master 127.9.93.62:38095 requested a full tablet report, sending...
I20260928 16:49:38.574043 10719 ts_manager.cc:194] Registered new tserver with Master: 9b6a6ab30f9d4ee1bc72adcb20bfecde (127.9.93.3:33469)
I20260928 16:49:38.574998 10719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.3@KRBTEST.COM'} at 127.9.93.3:53915
I20260928 16:49:38.586966 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 16:49:38 dist-test-slave-wzcb krb5kdc[10674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614178, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 16:49:38 dist-test-slave-wzcb krb5kdc[10674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614178, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:49:38.640388 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:38.624228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46456 (local address 127.9.93.62:38095)
0928 16:49:38.624480 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:38.624485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:38.624977 (+ 492us) server_negotiation.cc:407] Connection header received
0928 16:49:38.625122 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:38.625126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:38.625202 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:38.625308 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:38.626987 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.627943 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.629410 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.629953 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.630072 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:38.634678 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:38.634686 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:38.634690 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:38.634752 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:38.637615 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:38.638395 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:38.638400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:38.638402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:38.638560 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:38.638925 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:38.638927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:38.638929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:38.639234 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:38.639500 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:38.639808 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 16:49:38.640023 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260928 16:49:38.645828 10719 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:49:38.653785 10719 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:49:38.698975 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:38.669086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:33345 (local address 127.0.0.1:59596)
0928 16:49:38.686639 (+ 17553us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:38.686707 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:38.687264 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:38.688871 (+ 1607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:38.688880 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:38.688925 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:49:38.689510 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:38.689526 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.692186 (+ 2660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.692192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:38.695263 (+ 3071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:38.695281 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.697848 (+ 2567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.697853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:38.698059 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:38.698117 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 16:49:38.698261 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 16:49:38.698528 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17169,"mutex_wait_us":30,"thread_start_us":16963,"threads_started":1}
I20260928 16:49:38.699074 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:38.669373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59596 (local address 127.9.93.2:33345)
0928 16:49:38.684927 (+ 15554us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:38.684942 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:38.686934 (+ 1992us) server_negotiation.cc:407] Connection header received
0928 16:49:38.687655 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:38.687668 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:38.688187 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:38.688431 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:49:38.689774 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.691976 (+ 2202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.695537 (+ 3561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.697257 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.697417 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:38.697706 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:38.698415 (+ 709us) server_negotiation.cc:299] Negotiation successful
0928 16:49:38.698649 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15258,"thread_start_us":15585,"threads_started":1}
I20260928 16:49:38.700832 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:38.669424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:44601 (local address 127.0.0.1:38218)
0928 16:49:38.691092 (+ 21668us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:38.691118 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:38.691330 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:38.692338 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:38.692343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:38.692375 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:49:38.692878 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:38.692893 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.694981 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.694986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:38.697564 (+ 2578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:38.697581 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.699846 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.699855 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:38.700079 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:38.700115 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 16:49:38.700225 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 16:49:38.700405 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21487,"mutex_wait_us":5,"thread_start_us":22194,"threads_started":1}
I20260928 16:49:38.700837 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:38.670100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38218 (local address 127.9.93.1:44601)
0928 16:49:38.685760 (+ 15660us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:38.685772 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:38.691287 (+ 5515us) server_negotiation.cc:407] Connection header received
0928 16:49:38.691499 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:38.691509 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:38.691832 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:38.692083 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:49:38.693113 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.694784 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.697834 (+ 3050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.699584 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.699758 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:38.700021 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:38.700306 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 16:49:38.700507 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15467,"thread_start_us":15326,"threads_started":1}
I20260928 16:49:38.702265 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:38.670019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:33469 (local address 127.0.0.1:47940)
0928 16:49:38.691884 (+ 21865us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:38.691903 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:38.692018 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:38.693011 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:38.693015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:38.693042 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:49:38.693481 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:38.693491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.695105 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.695110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:38.697444 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:38.697460 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.700091 (+ 2631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.700099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:38.701492 (+ 1393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:38.701530 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 16:49:38.701634 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 16:49:38.701833 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21729,"mutex_wait_us":27,"thread_start_us":20717,"threads_started":1}
I20260928 16:49:38.702298 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:38.670829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47940 (local address 127.9.93.3:33469)
0928 16:49:38.690499 (+ 19670us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:38.690511 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:38.692169 (+ 1658us) server_negotiation.cc:407] Connection header received
0928 16:49:38.692313 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:38.692322 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:38.692630 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:38.692877 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:49:38.693728 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.694933 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.697651 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.699633 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.699773 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:38.701524 (+ 1751us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:38.701724 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 16:49:38.701942 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19415,"thread_start_us":19127,"threads_started":1}
I20260928 16:49:38.704447 10997 tablet_service.cc:1467] Processing CreateTablet for tablet 88ec2ebc8d2540a9869939514acf2cd5 (DEFAULT_TABLE table=test-table [id=35aa49f71cf84eccbe68d73a77732171]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:49:38.706023 10856 tablet_service.cc:1467] Processing CreateTablet for tablet 88ec2ebc8d2540a9869939514acf2cd5 (DEFAULT_TABLE table=test-table [id=35aa49f71cf84eccbe68d73a77732171]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:49:38.706697 10997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88ec2ebc8d2540a9869939514acf2cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:38.707533 10856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88ec2ebc8d2540a9869939514acf2cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:38.707141 11137 tablet_service.cc:1467] Processing CreateTablet for tablet 88ec2ebc8d2540a9869939514acf2cd5 (DEFAULT_TABLE table=test-table [id=35aa49f71cf84eccbe68d73a77732171]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:49:38.709005 11137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88ec2ebc8d2540a9869939514acf2cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:38.721699 11226 tablet_bootstrap.cc:492] T 88ec2ebc8d2540a9869939514acf2cd5 P 0afab931743a4802b0dfbcc0e95680b1: Bootstrap starting.
I20260928 16:49:38.725790 11226 tablet_bootstrap.cc:654] T 88ec2ebc8d2540a9869939514acf2cd5 P 0afab931743a4802b0dfbcc0e95680b1: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:38.726886 11228 tablet_bootstrap.cc:492] T 88ec2ebc8d2540a9869939514acf2cd5 P 9b6a6ab30f9d4ee1bc72adcb20bfecde: Bootstrap starting.
I20260928 16:49:38.727530 11226 log.cc:824] T 88ec2ebc8d2540a9869939514acf2cd5 P 0afab931743a4802b0dfbcc0e95680b1: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:38.732029 11228 tablet_bootstrap.cc:654] T 88ec2ebc8d2540a9869939514acf2cd5 P 9b6a6ab30f9d4ee1bc72adcb20bfecde: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:38.732724 11227 tablet_bootstrap.cc:492] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e: Bootstrap starting.
I20260928 16:49:38.734858 11226 tablet_bootstrap.cc:492] T 88ec2ebc8d2540a9869939514acf2cd5 P 0afab931743a4802b0dfbcc0e95680b1: No bootstrap required, opened a new log
I20260928 16:49:38.735179 11226 ts_tablet_manager.cc:1397] T 88ec2ebc8d2540a9869939514acf2cd5 P 0afab931743a4802b0dfbcc0e95680b1: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260928 16:49:38.735512 11228 log.cc:824] T 88ec2ebc8d2540a9869939514acf2cd5 P 9b6a6ab30f9d4ee1bc72adcb20bfecde: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:38.737257 11227 tablet_bootstrap.cc:654] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:38.738807 11227 log.cc:824] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:38.739686 11228 tablet_bootstrap.cc:492] T 88ec2ebc8d2540a9869939514acf2cd5 P 9b6a6ab30f9d4ee1bc72adcb20bfecde: No bootstrap required, opened a new log
I20260928 16:49:38.740010 11228 ts_tablet_manager.cc:1397] T 88ec2ebc8d2540a9869939514acf2cd5 P 9b6a6ab30f9d4ee1bc72adcb20bfecde: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260928 16:49:38.742769 11227 tablet_bootstrap.cc:492] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e: No bootstrap required, opened a new log
I20260928 16:49:38.743165 11227 ts_tablet_manager.cc:1397] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260928 16:49:38.743948 11226 raft_consensus.cc:348] T 88ec2ebc8d2540a9869939514acf2cd5 P 0afab931743a4802b0dfbcc0e95680b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:38.744530 11226 raft_consensus.cc:374] T 88ec2ebc8d2540a9869939514acf2cd5 P 0afab931743a4802b0dfbcc0e95680b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:38.744699 11226 raft_consensus.cc:729] T 88ec2ebc8d2540a9869939514acf2cd5 P 0afab931743a4802b0dfbcc0e95680b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0afab931743a4802b0dfbcc0e95680b1, State: Initialized, Role: FOLLOWER
I20260928 16:49:38.745657 11226 consensus_queue.cc:260] T 88ec2ebc8d2540a9869939514acf2cd5 P 0afab931743a4802b0dfbcc0e95680b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:38.748153 11226 ts_tablet_manager.cc:1428] T 88ec2ebc8d2540a9869939514acf2cd5 P 0afab931743a4802b0dfbcc0e95680b1: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260928 16:49:38.752061 11228 raft_consensus.cc:348] T 88ec2ebc8d2540a9869939514acf2cd5 P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:38.752647 11228 raft_consensus.cc:374] T 88ec2ebc8d2540a9869939514acf2cd5 P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:38.752775 11228 raft_consensus.cc:729] T 88ec2ebc8d2540a9869939514acf2cd5 P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b6a6ab30f9d4ee1bc72adcb20bfecde, State: Initialized, Role: FOLLOWER
I20260928 16:49:38.753533 11228 consensus_queue.cc:260] T 88ec2ebc8d2540a9869939514acf2cd5 P 9b6a6ab30f9d4ee1bc72adcb20bfecde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:38.755210 11207 heartbeater.cc:503] Master 127.9.93.62:38095 was elected leader, sending a full tablet report...
I20260928 16:49:38.754977 11227 raft_consensus.cc:348] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:38.755568 11227 raft_consensus.cc:374] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:38.755702 11227 raft_consensus.cc:729] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e320993c594fb5b5aae9684584cc6e, State: Initialized, Role: FOLLOWER
I20260928 16:49:38.756145 11228 ts_tablet_manager.cc:1428] T 88ec2ebc8d2540a9869939514acf2cd5 P 9b6a6ab30f9d4ee1bc72adcb20bfecde: Time spent starting tablet: real 0.016s user 0.008s sys 0.007s
I20260928 16:49:38.756456 11227 consensus_queue.cc:260] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:38.757696 11067 heartbeater.cc:503] Master 127.9.93.62:38095 was elected leader, sending a full tablet report...
I20260928 16:49:38.758437 11227 ts_tablet_manager.cc:1428] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e: Time spent starting tablet: real 0.015s user 0.009s sys 0.009s
W20260928 16:49:38.782423 10927 tablet.cc:2369] T 88ec2ebc8d2540a9869939514acf2cd5 P 0afab931743a4802b0dfbcc0e95680b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:49:38.818415 11208 tablet.cc:2369] T 88ec2ebc8d2540a9869939514acf2cd5 P 9b6a6ab30f9d4ee1bc72adcb20bfecde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:49:38.827116 11234 raft_consensus.cc:482] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:49:38.827644 11234 raft_consensus.cc:504] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:38.829928 11234 leader_election.cc:288] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0afab931743a4802b0dfbcc0e95680b1 (127.9.93.1:44601), 9b6a6ab30f9d4ee1bc72adcb20bfecde (127.9.93.3:33469)
I20260928 16:49:38.837344 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:38.830403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:40713 (local address 127.9.93.1:44601)
0928 16:49:38.830604 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:38.830610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:38.831394 (+ 784us) server_negotiation.cc:407] Connection header received
0928 16:49:38.831668 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:38.831673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:38.831760 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:38.831845 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:49:38.832484 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.833246 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.835365 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.836253 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.836405 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:38.836535 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:38.836918 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 16:49:38.837084 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:49:38.837460 11235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:38.830432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:44601 (local address 127.9.93.2:40713)
0928 16:49:38.831236 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:38.831257 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:38.831488 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:38.831887 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:38.831892 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:38.831921 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:49:38.832317 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:38.832328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.833383 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.833389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:38.835112 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:38.835131 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.836536 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.836543 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:38.836726 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:38.836756 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 16:49:38.836855 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 16:49:38.837046 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":690,"threads_started":1}
I20260928 16:49:38.837790 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:38.830430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:44845 (local address 127.9.93.3:33469)
0928 16:49:38.830656 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:38.830663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:38.831815 (+ 1152us) server_negotiation.cc:407] Connection header received
0928 16:49:38.831997 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:38.832002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:38.832081 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:38.832168 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:49:38.833002 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.833846 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.835758 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.836651 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.836839 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:38.837036 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:38.837354 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 16:49:38.837511 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:49:38.837838 11236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:38.830937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:33469 (local address 127.9.93.2:44845)
0928 16:49:38.831678 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:38.831694 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:38.831851 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:38.832214 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:38.832219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:38.832249 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:49:38.832785 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:38.832799 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.834001 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.834005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:38.835566 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:38.835581 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.836978 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.836984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:38.837174 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:38.837201 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 16:49:38.837297 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 16:49:38.837457 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":483,"threads_started":1}
I20260928 16:49:38.838977 10876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88ec2ebc8d2540a9869939514acf2cd5" candidate_uuid: "74e320993c594fb5b5aae9684584cc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0afab931743a4802b0dfbcc0e95680b1" is_pre_election: true
I20260928 16:49:38.838979 11162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88ec2ebc8d2540a9869939514acf2cd5" candidate_uuid: "74e320993c594fb5b5aae9684584cc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" is_pre_election: true
I20260928 16:49:38.839577 10876 raft_consensus.cc:2454] T 88ec2ebc8d2540a9869939514acf2cd5 P 0afab931743a4802b0dfbcc0e95680b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74e320993c594fb5b5aae9684584cc6e in term 0.
I20260928 16:49:38.839582 11162 raft_consensus.cc:2454] T 88ec2ebc8d2540a9869939514acf2cd5 P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74e320993c594fb5b5aae9684584cc6e in term 0.
I20260928 16:49:38.840380 10946 leader_election.cc:302] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74e320993c594fb5b5aae9684584cc6e, 9b6a6ab30f9d4ee1bc72adcb20bfecde; no voters:
I20260928 16:49:38.840979 11234 raft_consensus.cc:2781] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:49:38.841228 11234 raft_consensus.cc:482] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:49:38.841342 11234 raft_consensus.cc:3037] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:38.844002 11234 raft_consensus.cc:504] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:38.844741 11234 leader_election.cc:288] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e [CANDIDATE]: Term 1 election: Requested vote from peers 0afab931743a4802b0dfbcc0e95680b1 (127.9.93.1:44601), 9b6a6ab30f9d4ee1bc72adcb20bfecde (127.9.93.3:33469)
I20260928 16:49:38.845464 10876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88ec2ebc8d2540a9869939514acf2cd5" candidate_uuid: "74e320993c594fb5b5aae9684584cc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0afab931743a4802b0dfbcc0e95680b1"
I20260928 16:49:38.845465 11162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88ec2ebc8d2540a9869939514acf2cd5" candidate_uuid: "74e320993c594fb5b5aae9684584cc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde"
I20260928 16:49:38.845733 10876 raft_consensus.cc:3037] T 88ec2ebc8d2540a9869939514acf2cd5 P 0afab931743a4802b0dfbcc0e95680b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:38.845757 11162 raft_consensus.cc:3037] T 88ec2ebc8d2540a9869939514acf2cd5 P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:38.848455 10876 raft_consensus.cc:2454] T 88ec2ebc8d2540a9869939514acf2cd5 P 0afab931743a4802b0dfbcc0e95680b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74e320993c594fb5b5aae9684584cc6e in term 1.
I20260928 16:49:38.848455 11162 raft_consensus.cc:2454] T 88ec2ebc8d2540a9869939514acf2cd5 P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74e320993c594fb5b5aae9684584cc6e in term 1.
I20260928 16:49:38.849108 10947 leader_election.cc:302] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0afab931743a4802b0dfbcc0e95680b1, 74e320993c594fb5b5aae9684584cc6e, 9b6a6ab30f9d4ee1bc72adcb20bfecde; no voters:
I20260928 16:49:38.849529 11234 raft_consensus.cc:2781] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:38.850087 11234 raft_consensus.cc:686] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e [term 1 LEADER]: Becoming Leader. State: Replica: 74e320993c594fb5b5aae9684584cc6e, State: Running, Role: LEADER
I20260928 16:49:38.850813 11234 consensus_queue.cc:237] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:38.856213 10719 catalog_manager.cc:5495] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 74e320993c594fb5b5aae9684584cc6e (127.9.93.2). New cstate: current_term: 1 leader_uuid: "74e320993c594fb5b5aae9684584cc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } health_report { overall_health: UNKNOWN } } }
W20260928 16:49:38.913179 11068 tablet.cc:2369] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:49:38.918732 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:38.910842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59602 (local address 127.9.93.2:33345)
0928 16:49:38.911126 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:38.911135 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:38.911284 (+ 149us) server_negotiation.cc:407] Connection header received
0928 16:49:38.911785 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:38.911790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:38.911888 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:38.912059 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:38.912532 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.913724 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.915949 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:38.916454 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:38.916574 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:38.917267 (+ 693us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:38.918020 (+ 753us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:38.918164 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:38.918339 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 16:49:38.918442 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 16:49:38.930892 11162 raft_consensus.cc:1261] T 88ec2ebc8d2540a9869939514acf2cd5 P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 1 FOLLOWER]: Refusing update from remote peer 74e320993c594fb5b5aae9684584cc6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:49:38.930892 10876 raft_consensus.cc:1261] T 88ec2ebc8d2540a9869939514acf2cd5 P 0afab931743a4802b0dfbcc0e95680b1 [term 1 FOLLOWER]: Refusing update from remote peer 74e320993c594fb5b5aae9684584cc6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:49:38.932355 11237 consensus_queue.cc:1035] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:49:38.933992 11237 consensus_queue.cc:1035] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:49:38.975096 10718 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:49:38.975564 10718 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:49:38.980412 10718 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b45dbbb1946462a8537514ca7a9fc75: Sending DeleteTablet for 3 replicas of tablet 88ec2ebc8d2540a9869939514acf2cd5
I20260928 16:49:38.982017 11137 tablet_service.cc:1514] Processing DeleteTablet for tablet 88ec2ebc8d2540a9869939514acf2cd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:49:38 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:47940
I20260928 16:49:38.981990 10856 tablet_service.cc:1514] Processing DeleteTablet for tablet 88ec2ebc8d2540a9869939514acf2cd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:49:38 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:38218
I20260928 16:49:38.982129 10997 tablet_service.cc:1514] Processing DeleteTablet for tablet 88ec2ebc8d2540a9869939514acf2cd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:49:38 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:59596
I20260928 16:49:38.988010 10718 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:49:38.988900 10718 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:49:38.998517 11252 tablet_replica.cc:330] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e: stopping tablet replica
I20260928 16:49:39.000799 11252 raft_consensus.cc:2229] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:49:39.001550 10855 tablet_service.cc:1467] Processing CreateTablet for tablet fca7bea2895242d4b090142a5b7a2f5e (DEFAULT_TABLE table=test-table [id=792733a0e9ce441c8ef16539ff0c64e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:49:39.002296 10855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fca7bea2895242d4b090142a5b7a2f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:39.003172 11250 tablet_replica.cc:330] T 88ec2ebc8d2540a9869939514acf2cd5 P 0afab931743a4802b0dfbcc0e95680b1: stopping tablet replica
I20260928 16:49:39.003584 11252 raft_consensus.cc:2258] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:49:39.004143 11251 tablet_replica.cc:330] T 88ec2ebc8d2540a9869939514acf2cd5 P 9b6a6ab30f9d4ee1bc72adcb20bfecde: stopping tablet replica
I20260928 16:49:39.004595 11136 tablet_service.cc:1467] Processing CreateTablet for tablet fca7bea2895242d4b090142a5b7a2f5e (DEFAULT_TABLE table=test-table [id=792733a0e9ce441c8ef16539ff0c64e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:49:39.005061 11250 raft_consensus.cc:2229] T 88ec2ebc8d2540a9869939514acf2cd5 P 0afab931743a4802b0dfbcc0e95680b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:49:39.005299 11136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fca7bea2895242d4b090142a5b7a2f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:39.005689 11250 raft_consensus.cc:2258] T 88ec2ebc8d2540a9869939514acf2cd5 P 0afab931743a4802b0dfbcc0e95680b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:49:39.006992 11252 ts_tablet_manager.cc:1905] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:49:39.008312 10997 tablet_service.cc:1467] Processing CreateTablet for tablet fca7bea2895242d4b090142a5b7a2f5e (DEFAULT_TABLE table=test-table [id=792733a0e9ce441c8ef16539ff0c64e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:49:39.008517 11250 ts_tablet_manager.cc:1905] T 88ec2ebc8d2540a9869939514acf2cd5 P 0afab931743a4802b0dfbcc0e95680b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:49:39.008885 10997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fca7bea2895242d4b090142a5b7a2f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:39.016251 11252 ts_tablet_manager.cc:1918] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:49:39.016525 11250 ts_tablet_manager.cc:1918] T 88ec2ebc8d2540a9869939514acf2cd5 P 0afab931743a4802b0dfbcc0e95680b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:49:39.016613 11252 log.cc:1196] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/88ec2ebc8d2540a9869939514acf2cd5
I20260928 16:49:39.016868 11250 log.cc:1196] T 88ec2ebc8d2540a9869939514acf2cd5 P 0afab931743a4802b0dfbcc0e95680b1: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-0/wal/wals/88ec2ebc8d2540a9869939514acf2cd5
I20260928 16:49:39.017551 11250 ts_tablet_manager.cc:1939] T 88ec2ebc8d2540a9869939514acf2cd5 P 0afab931743a4802b0dfbcc0e95680b1: Deleting consensus metadata
I20260928 16:49:39.017568 11252 ts_tablet_manager.cc:1939] T 88ec2ebc8d2540a9869939514acf2cd5 P 74e320993c594fb5b5aae9684584cc6e: Deleting consensus metadata
I20260928 16:49:39.019912 10704 catalog_manager.cc:4841] TS 0afab931743a4802b0dfbcc0e95680b1 (127.9.93.1:44601): tablet 88ec2ebc8d2540a9869939514acf2cd5 (table test-table [id=35aa49f71cf84eccbe68d73a77732171]) successfully deleted
I20260928 16:49:39.019873 10706 catalog_manager.cc:4841] TS 74e320993c594fb5b5aae9684584cc6e (127.9.93.2:33345): tablet 88ec2ebc8d2540a9869939514acf2cd5 (table test-table [id=35aa49f71cf84eccbe68d73a77732171]) successfully deleted
I20260928 16:49:39.006273 11251 raft_consensus.cc:2229] T 88ec2ebc8d2540a9869939514acf2cd5 P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:49:39.021274 11251 raft_consensus.cc:2258] T 88ec2ebc8d2540a9869939514acf2cd5 P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:49:39.024345 11251 ts_tablet_manager.cc:1905] T 88ec2ebc8d2540a9869939514acf2cd5 P 9b6a6ab30f9d4ee1bc72adcb20bfecde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:49:39.024827 11227 tablet_bootstrap.cc:492] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e: Bootstrap starting.
I20260928 16:49:39.026858 11227 tablet_bootstrap.cc:654] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:39.029696 11226 tablet_bootstrap.cc:492] T fca7bea2895242d4b090142a5b7a2f5e P 0afab931743a4802b0dfbcc0e95680b1: Bootstrap starting.
I20260928 16:49:39.031497 11228 tablet_bootstrap.cc:492] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde: Bootstrap starting.
I20260928 16:49:39.031599 11226 tablet_bootstrap.cc:654] T fca7bea2895242d4b090142a5b7a2f5e P 0afab931743a4802b0dfbcc0e95680b1: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:39.034062 11228 tablet_bootstrap.cc:654] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:39.039402 11251 ts_tablet_manager.cc:1918] T 88ec2ebc8d2540a9869939514acf2cd5 P 9b6a6ab30f9d4ee1bc72adcb20bfecde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:49:39.040208 11251 log.cc:1196] T 88ec2ebc8d2540a9869939514acf2cd5 P 9b6a6ab30f9d4ee1bc72adcb20bfecde: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/88ec2ebc8d2540a9869939514acf2cd5
I20260928 16:49:39.040953 11226 tablet_bootstrap.cc:492] T fca7bea2895242d4b090142a5b7a2f5e P 0afab931743a4802b0dfbcc0e95680b1: No bootstrap required, opened a new log
I20260928 16:49:39.040978 11251 ts_tablet_manager.cc:1939] T 88ec2ebc8d2540a9869939514acf2cd5 P 9b6a6ab30f9d4ee1bc72adcb20bfecde: Deleting consensus metadata
I20260928 16:49:39.041174 11226 ts_tablet_manager.cc:1397] T fca7bea2895242d4b090142a5b7a2f5e P 0afab931743a4802b0dfbcc0e95680b1: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260928 16:49:39.042225 11226 raft_consensus.cc:348] T fca7bea2895242d4b090142a5b7a2f5e P 0afab931743a4802b0dfbcc0e95680b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:39.042536 11226 raft_consensus.cc:374] T fca7bea2895242d4b090142a5b7a2f5e P 0afab931743a4802b0dfbcc0e95680b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:39.042645 11226 raft_consensus.cc:729] T fca7bea2895242d4b090142a5b7a2f5e P 0afab931743a4802b0dfbcc0e95680b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0afab931743a4802b0dfbcc0e95680b1, State: Initialized, Role: FOLLOWER
I20260928 16:49:39.043012 11226 consensus_queue.cc:260] T fca7bea2895242d4b090142a5b7a2f5e P 0afab931743a4802b0dfbcc0e95680b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:39.043884 11226 ts_tablet_manager.cc:1428] T fca7bea2895242d4b090142a5b7a2f5e P 0afab931743a4802b0dfbcc0e95680b1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 16:49:39.043984 10703 catalog_manager.cc:4841] TS 9b6a6ab30f9d4ee1bc72adcb20bfecde (127.9.93.3:33469): tablet 88ec2ebc8d2540a9869939514acf2cd5 (table test-table [id=35aa49f71cf84eccbe68d73a77732171]) successfully deleted
I20260928 16:49:39.045950 11227 tablet_bootstrap.cc:492] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e: No bootstrap required, opened a new log
I20260928 16:49:39.046186 11227 ts_tablet_manager.cc:1397] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.012s
I20260928 16:49:39.047340 11228 tablet_bootstrap.cc:492] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde: No bootstrap required, opened a new log
I20260928 16:49:39.047616 11228 ts_tablet_manager.cc:1397] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.006s
I20260928 16:49:39.047474 11227 raft_consensus.cc:348] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:39.047788 11227 raft_consensus.cc:374] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:39.047914 11227 raft_consensus.cc:729] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e320993c594fb5b5aae9684584cc6e, State: Initialized, Role: FOLLOWER
I20260928 16:49:39.048260 11227 consensus_queue.cc:260] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:39.049139 11227 ts_tablet_manager.cc:1428] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 16:49:39.048974 11228 raft_consensus.cc:348] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:39.049335 11228 raft_consensus.cc:374] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:39.049491 11228 raft_consensus.cc:729] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b6a6ab30f9d4ee1bc72adcb20bfecde, State: Initialized, Role: FOLLOWER
I20260928 16:49:39.049793 11228 consensus_queue.cc:260] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:39.050520 11228 ts_tablet_manager.cc:1428] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260928 16:49:39.075979 11233 raft_consensus.cc:482] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:49:39.076241 11233 raft_consensus.cc:504] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:39.078017 11233 leader_election.cc:288] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0afab931743a4802b0dfbcc0e95680b1 (127.9.93.1:44601), 74e320993c594fb5b5aae9684584cc6e (127.9.93.2:33345)
I20260928 16:49:39.086983 11245 raft_consensus.cc:482] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:49:39.087296 11245 raft_consensus.cc:504] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:39.088392 11245 leader_election.cc:288] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0afab931743a4802b0dfbcc0e95680b1 (127.9.93.1:44601), 9b6a6ab30f9d4ee1bc72adcb20bfecde (127.9.93.3:33469)
I20260928 16:49:39.088644 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:39.078958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:43579 (local address 127.9.93.1:44601)
0928 16:49:39.079161 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:39.079168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:39.080163 (+ 995us) server_negotiation.cc:407] Connection header received
0928 16:49:39.080446 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:39.080454 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:39.080564 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:39.080668 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:49:39.081438 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.082612 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.085416 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.086727 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.086855 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:39.086963 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:39.088158 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0928 16:49:39.088340 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 16:49:39.089298 10876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fca7bea2895242d4b090142a5b7a2f5e" candidate_uuid: "74e320993c594fb5b5aae9684584cc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0afab931743a4802b0dfbcc0e95680b1" is_pre_election: true
I20260928 16:49:39.089315 11162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fca7bea2895242d4b090142a5b7a2f5e" candidate_uuid: "74e320993c594fb5b5aae9684584cc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" is_pre_election: true
I20260928 16:49:39.089591 10876 raft_consensus.cc:2454] T fca7bea2895242d4b090142a5b7a2f5e P 0afab931743a4802b0dfbcc0e95680b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74e320993c594fb5b5aae9684584cc6e in term 0.
I20260928 16:49:39.089609 11162 raft_consensus.cc:2454] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74e320993c594fb5b5aae9684584cc6e in term 0.
I20260928 16:49:39.090061 10947 leader_election.cc:302] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0afab931743a4802b0dfbcc0e95680b1, 74e320993c594fb5b5aae9684584cc6e; no voters:
I20260928 16:49:39.090312 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:39.078691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:49899 (local address 127.9.93.2:33345)
0928 16:49:39.078980 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:39.078988 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:39.080880 (+ 1892us) server_negotiation.cc:407] Connection header received
0928 16:49:39.081037 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:39.081041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:39.081120 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:39.081210 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:49:39.081758 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.082781 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.085018 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.086396 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.086572 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:39.086703 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:39.089369 (+ 2666us) server_negotiation.cc:299] Negotiation successful
0928 16:49:39.089662 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260928 16:49:39.090669 11245 raft_consensus.cc:2781] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:49:39.090821 11245 raft_consensus.cc:482] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:49:39.090927 11245 raft_consensus.cc:3037] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:39.091095 11257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:39.078390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:33345 (local address 127.9.93.3:49899)
0928 16:49:39.080762 (+ 2372us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:39.080777 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:39.080894 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:39.081243 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:39.081247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:39.081277 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:49:39.081573 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:39.081583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.082947 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.082956 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:39.084807 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:39.084835 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.087108 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.087116 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:39.088650 (+ 1534us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:39.088685 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:49:39.090095 (+ 1410us) client_negotiation.cc:240] Negotiation successful
0928 16:49:39.090650 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2264,"mutex_wait_us":96,"thread_start_us":2063,"threads_started":1}
I20260928 16:49:39.091496 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:39.078377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:44601 (local address 127.9.93.3:43579)
0928 16:49:39.079978 (+ 1601us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:39.080007 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:39.080243 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:39.080867 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:39.080873 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:39.080904 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:49:39.081221 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:39.081234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.082819 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.082825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:39.085197 (+ 2372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:39.085214 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.087589 (+ 2375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.087597 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:39.087922 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:39.087962 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 16:49:39.088083 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 16:49:39.088258 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1347,"mutex_wait_us":24,"thread_start_us":835,"threads_started":1}
I20260928 16:49:39.092367 10876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fca7bea2895242d4b090142a5b7a2f5e" candidate_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0afab931743a4802b0dfbcc0e95680b1" is_pre_election: true
I20260928 16:49:39.092657 10876 raft_consensus.cc:2454] T fca7bea2895242d4b090142a5b7a2f5e P 0afab931743a4802b0dfbcc0e95680b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b6a6ab30f9d4ee1bc72adcb20bfecde in term 0.
I20260928 16:49:39.092736 11017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fca7bea2895242d4b090142a5b7a2f5e" candidate_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74e320993c594fb5b5aae9684584cc6e" is_pre_election: true
I20260928 16:49:39.093400 11088 leader_election.cc:302] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0afab931743a4802b0dfbcc0e95680b1, 9b6a6ab30f9d4ee1bc72adcb20bfecde; no voters:
I20260928 16:49:39.093950 11233 raft_consensus.cc:2781] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:49:39.094106 11233 raft_consensus.cc:482] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:49:39.094223 11233 raft_consensus.cc:3037] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:39.094354 11245 raft_consensus.cc:504] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:39.095086 11017 raft_consensus.cc:2379] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9b6a6ab30f9d4ee1bc72adcb20bfecde in current term 1: Already voted for candidate 74e320993c594fb5b5aae9684584cc6e in this term.
I20260928 16:49:39.095767 10876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fca7bea2895242d4b090142a5b7a2f5e" candidate_uuid: "74e320993c594fb5b5aae9684584cc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0afab931743a4802b0dfbcc0e95680b1"
I20260928 16:49:39.095996 10876 raft_consensus.cc:3037] T fca7bea2895242d4b090142a5b7a2f5e P 0afab931743a4802b0dfbcc0e95680b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:39.098788 11233 raft_consensus.cc:504] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:39.099370 10876 raft_consensus.cc:2454] T fca7bea2895242d4b090142a5b7a2f5e P 0afab931743a4802b0dfbcc0e95680b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74e320993c594fb5b5aae9684584cc6e in term 1.
I20260928 16:49:39.099716 11245 leader_election.cc:288] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e [CANDIDATE]: Term 1 election: Requested vote from peers 0afab931743a4802b0dfbcc0e95680b1 (127.9.93.1:44601), 9b6a6ab30f9d4ee1bc72adcb20bfecde (127.9.93.3:33469)
I20260928 16:49:39.100036 10947 leader_election.cc:302] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0afab931743a4802b0dfbcc0e95680b1, 74e320993c594fb5b5aae9684584cc6e; no voters:
I20260928 16:49:39.100373 11245 raft_consensus.cc:2781] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:39.100345 10875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fca7bea2895242d4b090142a5b7a2f5e" candidate_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0afab931743a4802b0dfbcc0e95680b1"
I20260928 16:49:39.100659 11245 raft_consensus.cc:686] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e [term 1 LEADER]: Becoming Leader. State: Replica: 74e320993c594fb5b5aae9684584cc6e, State: Running, Role: LEADER
I20260928 16:49:39.100813 10875 raft_consensus.cc:2379] T fca7bea2895242d4b090142a5b7a2f5e P 0afab931743a4802b0dfbcc0e95680b1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9b6a6ab30f9d4ee1bc72adcb20bfecde in current term 1: Already voted for candidate 74e320993c594fb5b5aae9684584cc6e in this term.
I20260928 16:49:39.100773 11162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fca7bea2895242d4b090142a5b7a2f5e" candidate_uuid: "74e320993c594fb5b5aae9684584cc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde"
I20260928 16:49:39.101125 11162 raft_consensus.cc:2379] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 74e320993c594fb5b5aae9684584cc6e in current term 1: Already voted for candidate 9b6a6ab30f9d4ee1bc72adcb20bfecde in this term.
I20260928 16:49:39.101267 11245 consensus_queue.cc:237] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:39.101508 11017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fca7bea2895242d4b090142a5b7a2f5e" candidate_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74e320993c594fb5b5aae9684584cc6e"
I20260928 16:49:39.102025 11233 leader_election.cc:288] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde [CANDIDATE]: Term 1 election: Requested vote from peers 0afab931743a4802b0dfbcc0e95680b1 (127.9.93.1:44601), 74e320993c594fb5b5aae9684584cc6e (127.9.93.2:33345)
I20260928 16:49:39.102321 11090 leader_election.cc:302] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b6a6ab30f9d4ee1bc72adcb20bfecde; no voters: 0afab931743a4802b0dfbcc0e95680b1, 74e320993c594fb5b5aae9684584cc6e
I20260928 16:49:39.102852 11233 raft_consensus.cc:2726] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 16:49:39.105537 10717 catalog_manager.cc:5495] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 74e320993c594fb5b5aae9684584cc6e (127.9.93.2). New cstate: current_term: 1 leader_uuid: "74e320993c594fb5b5aae9684584cc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } health_report { overall_health: UNKNOWN } } }
Sep 28 16:49:39 dist-test-slave-wzcb krb5kdc[10674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614176, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:49:39.174074 11264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:39.158052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:38095 (local address 127.0.0.1:46458)
0928 16:49:39.158850 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:39.158880 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:39.159101 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:39.159547 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:39.159553 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:39.160433 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:39.160737 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:39.160748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.162096 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.162101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:39.163149 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:39.163176 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.164024 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.164032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:39.164237 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:39.164983 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:39.165003 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:39.167758 (+ 2755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:39.171160 (+ 3402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:39.171181 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:39.171191 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:39.171792 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:39.172205 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:39.172210 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:39.172214 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:39.172374 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:39.172937 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:39.172947 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:39.173152 (+ 205us) client_negotiation.cc:769] Sending connection context
0928 16:49:39.173311 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 16:49:39.173642 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":349,"threads_started":1}
I20260928 16:49:39.174197 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:39.158106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46458 (local address 127.9.93.62:38095)
0928 16:49:39.159210 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:39.159215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:39.159238 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:49:39.159281 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:39.159284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:39.159389 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:39.159499 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:39.161022 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.161943 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.163355 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.163813 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.163934 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:39.168026 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:39.168031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:39.168035 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:39.168073 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:39.170973 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:39.171927 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:39.171933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:39.171935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:39.172088 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:39.172483 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:39.172487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:39.172489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:39.172795 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:39.172962 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:39.173421 (+ 459us) server_negotiation.cc:299] Negotiation successful
0928 16:49:39.173816 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":153,"server-negotiator.queue_time_us":972,"thread_start_us":811,"threads_started":1}
I20260928 16:49:39.180519 10717 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:46458:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 16:49:39.182037 10717 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:49:39.190086 10855 tablet_service.cc:1467] Processing CreateTablet for tablet 2304b864b5c142879203bf725cd4a462 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d5d97f4bb874d7d9a7a4320574bf90f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:49:39.190793 10855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2304b864b5c142879203bf725cd4a462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:39.192013 10997 tablet_service.cc:1467] Processing CreateTablet for tablet 2304b864b5c142879203bf725cd4a462 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d5d97f4bb874d7d9a7a4320574bf90f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:49:39.192775 10997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2304b864b5c142879203bf725cd4a462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:39.196494 11136 tablet_service.cc:1467] Processing CreateTablet for tablet 2304b864b5c142879203bf725cd4a462 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d5d97f4bb874d7d9a7a4320574bf90f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:49:39.197242 11136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2304b864b5c142879203bf725cd4a462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:39.202802 11227 tablet_bootstrap.cc:492] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e: Bootstrap starting.
I20260928 16:49:39.204972 11227 tablet_bootstrap.cc:654] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:39.209619 11228 tablet_bootstrap.cc:492] T 2304b864b5c142879203bf725cd4a462 P 9b6a6ab30f9d4ee1bc72adcb20bfecde: Bootstrap starting.
I20260928 16:49:39.211880 11228 tablet_bootstrap.cc:654] T 2304b864b5c142879203bf725cd4a462 P 9b6a6ab30f9d4ee1bc72adcb20bfecde: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:39.212935 11227 tablet_bootstrap.cc:492] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e: No bootstrap required, opened a new log
I20260928 16:49:39.213299 11227 ts_tablet_manager.cc:1397] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.001s
I20260928 16:49:39.214353 11227 raft_consensus.cc:348] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:39.214640 11227 raft_consensus.cc:374] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:39.214684 11228 tablet_bootstrap.cc:492] T 2304b864b5c142879203bf725cd4a462 P 9b6a6ab30f9d4ee1bc72adcb20bfecde: No bootstrap required, opened a new log
I20260928 16:49:39.214740 11227 raft_consensus.cc:729] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e320993c594fb5b5aae9684584cc6e, State: Initialized, Role: FOLLOWER
I20260928 16:49:39.214886 11228 ts_tablet_manager.cc:1397] T 2304b864b5c142879203bf725cd4a462 P 9b6a6ab30f9d4ee1bc72adcb20bfecde: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 16:49:39.215131 11227 consensus_queue.cc:260] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:39.215991 11227 ts_tablet_manager.cc:1428] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 16:49:39.215844 11245 tablet_replica.cc:439] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } } }
I20260928 16:49:39.216253 11245 tablet_replica.cc:442] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:49:39.217469 11228 raft_consensus.cc:348] T 2304b864b5c142879203bf725cd4a462 P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:39.217824 11228 raft_consensus.cc:374] T 2304b864b5c142879203bf725cd4a462 P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:39.217947 11228 raft_consensus.cc:729] T 2304b864b5c142879203bf725cd4a462 P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b6a6ab30f9d4ee1bc72adcb20bfecde, State: Initialized, Role: FOLLOWER
I20260928 16:49:39.218281 11228 consensus_queue.cc:260] T 2304b864b5c142879203bf725cd4a462 P 9b6a6ab30f9d4ee1bc72adcb20bfecde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:39.218835 11233 tablet_replica.cc:439] T 2304b864b5c142879203bf725cd4a462 P 9b6a6ab30f9d4ee1bc72adcb20bfecde: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } } }
I20260928 16:49:39.219079 11228 ts_tablet_manager.cc:1428] T 2304b864b5c142879203bf725cd4a462 P 9b6a6ab30f9d4ee1bc72adcb20bfecde: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 16:49:39.219197 11233 tablet_replica.cc:442] T 2304b864b5c142879203bf725cd4a462 P 9b6a6ab30f9d4ee1bc72adcb20bfecde: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:49:39.221644 11226 tablet_bootstrap.cc:492] T 2304b864b5c142879203bf725cd4a462 P 0afab931743a4802b0dfbcc0e95680b1: Bootstrap starting.
I20260928 16:49:39.224023 11226 tablet_bootstrap.cc:654] T 2304b864b5c142879203bf725cd4a462 P 0afab931743a4802b0dfbcc0e95680b1: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:39.226423 11226 tablet_bootstrap.cc:492] T 2304b864b5c142879203bf725cd4a462 P 0afab931743a4802b0dfbcc0e95680b1: No bootstrap required, opened a new log
I20260928 16:49:39.226614 11226 ts_tablet_manager.cc:1397] T 2304b864b5c142879203bf725cd4a462 P 0afab931743a4802b0dfbcc0e95680b1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 16:49:39.227332 11226 raft_consensus.cc:348] T 2304b864b5c142879203bf725cd4a462 P 0afab931743a4802b0dfbcc0e95680b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:39.227624 11226 raft_consensus.cc:374] T 2304b864b5c142879203bf725cd4a462 P 0afab931743a4802b0dfbcc0e95680b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:39.227718 11226 raft_consensus.cc:729] T 2304b864b5c142879203bf725cd4a462 P 0afab931743a4802b0dfbcc0e95680b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0afab931743a4802b0dfbcc0e95680b1, State: Initialized, Role: FOLLOWER
I20260928 16:49:39.227972 11226 consensus_queue.cc:260] T 2304b864b5c142879203bf725cd4a462 P 0afab931743a4802b0dfbcc0e95680b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:39.228636 11226 ts_tablet_manager.cc:1428] T 2304b864b5c142879203bf725cd4a462 P 0afab931743a4802b0dfbcc0e95680b1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 16:49:39.228509 11232 tablet_replica.cc:439] T 2304b864b5c142879203bf725cd4a462 P 0afab931743a4802b0dfbcc0e95680b1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } } }
I20260928 16:49:39.228931 11232 tablet_replica.cc:442] T 2304b864b5c142879203bf725cd4a462 P 0afab931743a4802b0dfbcc0e95680b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:49:39.241992 11245 raft_consensus.cc:482] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:49:39.242297 11245 raft_consensus.cc:504] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:39.243165 11245 leader_election.cc:288] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0afab931743a4802b0dfbcc0e95680b1 (127.9.93.1:44601), 9b6a6ab30f9d4ee1bc72adcb20bfecde (127.9.93.3:33469)
I20260928 16:49:39.243933 11162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2304b864b5c142879203bf725cd4a462" candidate_uuid: "74e320993c594fb5b5aae9684584cc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" is_pre_election: true
I20260928 16:49:39.243918 10876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2304b864b5c142879203bf725cd4a462" candidate_uuid: "74e320993c594fb5b5aae9684584cc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0afab931743a4802b0dfbcc0e95680b1" is_pre_election: true
I20260928 16:49:39.244230 11162 raft_consensus.cc:2454] T 2304b864b5c142879203bf725cd4a462 P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74e320993c594fb5b5aae9684584cc6e in term 0.
I20260928 16:49:39.244254 10876 raft_consensus.cc:2454] T 2304b864b5c142879203bf725cd4a462 P 0afab931743a4802b0dfbcc0e95680b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74e320993c594fb5b5aae9684584cc6e in term 0.
I20260928 16:49:39.244794 10946 leader_election.cc:302] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74e320993c594fb5b5aae9684584cc6e, 9b6a6ab30f9d4ee1bc72adcb20bfecde; no voters:
I20260928 16:49:39.245206 11245 raft_consensus.cc:2781] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:49:39.245321 11245 raft_consensus.cc:482] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:49:39.245479 11245 raft_consensus.cc:3037] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:39.248661 11245 raft_consensus.cc:504] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:39.249526 11245 leader_election.cc:288] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e [CANDIDATE]: Term 1 election: Requested vote from peers 0afab931743a4802b0dfbcc0e95680b1 (127.9.93.1:44601), 9b6a6ab30f9d4ee1bc72adcb20bfecde (127.9.93.3:33469)
I20260928 16:49:39.250228 10876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2304b864b5c142879203bf725cd4a462" candidate_uuid: "74e320993c594fb5b5aae9684584cc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0afab931743a4802b0dfbcc0e95680b1"
I20260928 16:49:39.250253 11162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2304b864b5c142879203bf725cd4a462" candidate_uuid: "74e320993c594fb5b5aae9684584cc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde"
I20260928 16:49:39.250528 10876 raft_consensus.cc:3037] T 2304b864b5c142879203bf725cd4a462 P 0afab931743a4802b0dfbcc0e95680b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:39.250535 11162 raft_consensus.cc:3037] T 2304b864b5c142879203bf725cd4a462 P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:39.252995 10876 raft_consensus.cc:2454] T 2304b864b5c142879203bf725cd4a462 P 0afab931743a4802b0dfbcc0e95680b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74e320993c594fb5b5aae9684584cc6e in term 1.
I20260928 16:49:39.253603 10947 leader_election.cc:302] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0afab931743a4802b0dfbcc0e95680b1, 74e320993c594fb5b5aae9684584cc6e; no voters:
I20260928 16:49:39.253958 11245 raft_consensus.cc:2781] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:39.253913 11162 raft_consensus.cc:2454] T 2304b864b5c142879203bf725cd4a462 P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74e320993c594fb5b5aae9684584cc6e in term 1.
I20260928 16:49:39.254339 11245 raft_consensus.cc:686] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e [term 1 LEADER]: Becoming Leader. State: Replica: 74e320993c594fb5b5aae9684584cc6e, State: Running, Role: LEADER
I20260928 16:49:39.254690 11245 consensus_queue.cc:237] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } }
I20260928 16:49:39.256851 11237 tablet_replica.cc:439] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e: TxnStatusTablet state changed. Reason: New leader 74e320993c594fb5b5aae9684584cc6e. Latest consensus state: current_term: 1 leader_uuid: "74e320993c594fb5b5aae9684584cc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } } }
I20260928 16:49:39.257274 11237 tablet_replica.cc:442] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:49:39.258795 11270 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 16:49:39.259186 11270 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 16:49:39.259790 10717 catalog_manager.cc:5495] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 74e320993c594fb5b5aae9684584cc6e (127.9.93.2). New cstate: current_term: 1 leader_uuid: "74e320993c594fb5b5aae9684584cc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } health_report { overall_health: UNKNOWN } } }
I20260928 16:49:39.287961 11264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:39.281684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:33345 (local address 127.0.0.1:59618)
0928 16:49:39.282022 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:39.282042 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:39.282181 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:39.282845 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:39.282850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:39.282863 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:49:39.283060 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:39.283069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.284430 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.284435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:39.285510 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:39.285525 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.286504 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.286511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:39.286715 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:39.286831 (+ 116us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:49:39.287332 (+ 501us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:49:39.287372 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 16:49:39.287446 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 16:49:39.287646 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260928 16:49:39.287963 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:39.281757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59618 (local address 127.9.93.2:33345)
0928 16:49:39.282501 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:39.282506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:39.282524 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:49:39.282600 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:39.282604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:39.282684 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:39.282770 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:39.283225 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.284274 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.285685 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.286312 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.286437 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:39.287032 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:39.287201 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:39.287285 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:39.287496 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 16:49:39.287610 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565}
I20260928 16:49:39.292927 10876 raft_consensus.cc:1261] T 2304b864b5c142879203bf725cd4a462 P 0afab931743a4802b0dfbcc0e95680b1 [term 1 FOLLOWER]: Refusing update from remote peer 74e320993c594fb5b5aae9684584cc6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:49:39.292958 11162 raft_consensus.cc:1261] T 2304b864b5c142879203bf725cd4a462 P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 1 FOLLOWER]: Refusing update from remote peer 74e320993c594fb5b5aae9684584cc6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:49:39.293799 11237 consensus_queue.cc:1035] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:49:39.294420 11245 consensus_queue.cc:1035] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:49:39.304736 11233 tablet_replica.cc:439] T 2304b864b5c142879203bf725cd4a462 P 9b6a6ab30f9d4ee1bc72adcb20bfecde: TxnStatusTablet state changed. Reason: New leader 74e320993c594fb5b5aae9684584cc6e. Latest consensus state: current_term: 1 leader_uuid: "74e320993c594fb5b5aae9684584cc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } } }
I20260928 16:49:39.305282 11233 tablet_replica.cc:442] T 2304b864b5c142879203bf725cd4a462 P 9b6a6ab30f9d4ee1bc72adcb20bfecde: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:49:39.311091 11232 tablet_replica.cc:439] T 2304b864b5c142879203bf725cd4a462 P 0afab931743a4802b0dfbcc0e95680b1: TxnStatusTablet state changed. Reason: New leader 74e320993c594fb5b5aae9684584cc6e. Latest consensus state: current_term: 1 leader_uuid: "74e320993c594fb5b5aae9684584cc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } } }
I20260928 16:49:39.311491 11232 tablet_replica.cc:442] T 2304b864b5c142879203bf725cd4a462 P 0afab931743a4802b0dfbcc0e95680b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:49:39.311476 11234 tablet_replica.cc:439] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74e320993c594fb5b5aae9684584cc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } } }
I20260928 16:49:39.311856 11234 tablet_replica.cc:442] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:49:39.312084 11240 mvcc.cc:205] Tried to move back new op lower bound from 7334355678375956480 to 7334355678230278144. Current Snapshot: MvccSnapshot[applied={T|T < 7334355678375956480}]
I20260928 16:49:39.317301 11233 tablet_replica.cc:439] T 2304b864b5c142879203bf725cd4a462 P 9b6a6ab30f9d4ee1bc72adcb20bfecde: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74e320993c594fb5b5aae9684584cc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } } }
I20260928 16:49:39.317662 11233 tablet_replica.cc:442] T 2304b864b5c142879203bf725cd4a462 P 9b6a6ab30f9d4ee1bc72adcb20bfecde: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:49:39.318533 11237 tablet_replica.cc:439] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74e320993c594fb5b5aae9684584cc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } } }
I20260928 16:49:39.318907 11237 tablet_replica.cc:442] T 2304b864b5c142879203bf725cd4a462 P 74e320993c594fb5b5aae9684584cc6e: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:49:39.320859 11232 tablet_replica.cc:439] T 2304b864b5c142879203bf725cd4a462 P 0afab931743a4802b0dfbcc0e95680b1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74e320993c594fb5b5aae9684584cc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 } } peers { permanent_uuid: "74e320993c594fb5b5aae9684584cc6e" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33345 } } peers { permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 } } }
I20260928 16:49:39.321147 11232 tablet_replica.cc:442] T 2304b864b5c142879203bf725cd4a462 P 0afab931743a4802b0dfbcc0e95680b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:49:39.344590 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:39.338919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:35881 (local address 127.9.93.2:33345)
0928 16:49:39.339099 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:39.339106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:39.339190 (+ 84us) server_negotiation.cc:407] Connection header received
0928 16:49:39.339483 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:39.339488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:39.339564 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:39.339645 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:39.340045 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.340915 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.342566 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.343045 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.343165 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:39.343719 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:39.343864 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:39.343950 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:39.344072 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 16:49:39.344203 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":53}
I20260928 16:49:39.344702 11276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:39.338375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:33345 (local address 127.9.93.2:35881)
0928 16:49:39.339071 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:39.339098 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:39.339295 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:39.339675 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:39.339679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:39.339693 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:49:39.339914 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:39.339925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.341052 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.341058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:39.342385 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:39.342405 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.343229 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.343236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:39.343451 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:39.343536 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:49:39.343956 (+ 420us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:49:39.343971 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 16:49:39.344053 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 16:49:39.344251 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":1,"thread_start_us":312,"threads_started":1}
I20260928 16:49:39.358561 10876 raft_consensus.cc:1261] T fca7bea2895242d4b090142a5b7a2f5e P 0afab931743a4802b0dfbcc0e95680b1 [term 1 FOLLOWER]: Refusing update from remote peer 74e320993c594fb5b5aae9684584cc6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:49:39.359019 11162 raft_consensus.cc:1261] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 1 FOLLOWER]: Refusing update from remote peer 74e320993c594fb5b5aae9684584cc6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:49:39.359344 11237 consensus_queue.cc:1035] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afab931743a4802b0dfbcc0e95680b1" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:49:39.359706 11245 consensus_queue.cc:1035] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b6a6ab30f9d4ee1bc72adcb20bfecde" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 33469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:49:39.445003 10719 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:49:39.445288 10719 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:49:39.449129 10719 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b45dbbb1946462a8537514ca7a9fc75: Sending DeleteTablet for 3 replicas of tablet fca7bea2895242d4b090142a5b7a2f5e
I20260928 16:49:39.450191 10855 tablet_service.cc:1514] Processing DeleteTablet for tablet fca7bea2895242d4b090142a5b7a2f5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:49:39 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:38218
I20260928 16:49:39.450413 10997 tablet_service.cc:1514] Processing DeleteTablet for tablet fca7bea2895242d4b090142a5b7a2f5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:49:39 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:59596
I20260928 16:49:39.450644 11136 tablet_service.cc:1514] Processing DeleteTablet for tablet fca7bea2895242d4b090142a5b7a2f5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:49:39 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:47940
I20260928 16:49:39.450815 11252 tablet_replica.cc:330] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e: stopping tablet replica
I20260928 16:49:39.451022 11251 tablet_replica.cc:330] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde: stopping tablet replica
I20260928 16:49:39.451043 11250 tablet_replica.cc:330] T fca7bea2895242d4b090142a5b7a2f5e P 0afab931743a4802b0dfbcc0e95680b1: stopping tablet replica
I20260928 16:49:39.451076 11252 raft_consensus.cc:2229] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:49:39.451268 11251 raft_consensus.cc:2229] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:49:39.451326 11250 raft_consensus.cc:2229] T fca7bea2895242d4b090142a5b7a2f5e P 0afab931743a4802b0dfbcc0e95680b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:49:39.451555 11250 raft_consensus.cc:2258] T fca7bea2895242d4b090142a5b7a2f5e P 0afab931743a4802b0dfbcc0e95680b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:49:39.451556 11251 raft_consensus.cc:2258] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:49:39.451594 11252 raft_consensus.cc:2258] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:49:39.453493 11252 ts_tablet_manager.cc:1905] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:49:39.453490 11251 ts_tablet_manager.cc:1905] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:49:39.454780 11250 ts_tablet_manager.cc:1905] T fca7bea2895242d4b090142a5b7a2f5e P 0afab931743a4802b0dfbcc0e95680b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:49:39.462159 11251 ts_tablet_manager.cc:1918] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:49:39.462402 11251 log.cc:1196] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/fca7bea2895242d4b090142a5b7a2f5e
I20260928 16:49:39.462533 11250 ts_tablet_manager.cc:1918] T fca7bea2895242d4b090142a5b7a2f5e P 0afab931743a4802b0dfbcc0e95680b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:49:39.462725 11250 log.cc:1196] T fca7bea2895242d4b090142a5b7a2f5e P 0afab931743a4802b0dfbcc0e95680b1: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-0/wal/wals/fca7bea2895242d4b090142a5b7a2f5e
I20260928 16:49:39.462888 11251 ts_tablet_manager.cc:1939] T fca7bea2895242d4b090142a5b7a2f5e P 9b6a6ab30f9d4ee1bc72adcb20bfecde: Deleting consensus metadata
I20260928 16:49:39.462909 11252 ts_tablet_manager.cc:1918] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:49:39.463219 11250 ts_tablet_manager.cc:1939] T fca7bea2895242d4b090142a5b7a2f5e P 0afab931743a4802b0dfbcc0e95680b1: Deleting consensus metadata
I20260928 16:49:39.463217 11252 log.cc:1196] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/fca7bea2895242d4b090142a5b7a2f5e
I20260928 16:49:39.463846 11252 ts_tablet_manager.cc:1939] T fca7bea2895242d4b090142a5b7a2f5e P 74e320993c594fb5b5aae9684584cc6e: Deleting consensus metadata
I20260928 16:49:39.464542 10703 catalog_manager.cc:4841] TS 9b6a6ab30f9d4ee1bc72adcb20bfecde (127.9.93.3:33469): tablet fca7bea2895242d4b090142a5b7a2f5e (table test-table [id=792733a0e9ce441c8ef16539ff0c64e7]) successfully deleted
I20260928 16:49:39.465410 10706 catalog_manager.cc:4841] TS 74e320993c594fb5b5aae9684584cc6e (127.9.93.2:33345): tablet fca7bea2895242d4b090142a5b7a2f5e (table test-table [id=792733a0e9ce441c8ef16539ff0c64e7]) successfully deleted
I20260928 16:49:39.467211 10704 catalog_manager.cc:4841] TS 0afab931743a4802b0dfbcc0e95680b1 (127.9.93.1:44601): tablet fca7bea2895242d4b090142a5b7a2f5e (table test-table [id=792733a0e9ce441c8ef16539ff0c64e7]) successfully deleted
Sep 28 16:49:39 dist-test-slave-wzcb krb5kdc[10674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614178, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.93.1@KRBTEST.COM
I20260928 16:49:39.484359 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:39.464959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38224 (local address 127.9.93.1:44601)
0928 16:49:39.465202 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:39.465211 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:39.466246 (+ 1035us) server_negotiation.cc:407] Connection header received
0928 16:49:39.466448 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:39.466454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:39.466560 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:39.466693 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:39.468523 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.469708 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.471464 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.472215 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.472383 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:39.477220 (+ 4837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:39.477249 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:39.477274 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:39.477353 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:39.480905 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:39.481847 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:39.481857 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:39.481860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:39.482132 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:39.482573 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:39.482577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:39.482580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:39.483007 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:39.483398 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:39.483725 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 16:49:39.483975 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260928 16:49:39.485260 10921 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38224
I20260928 16:49:39.503197 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:39.488548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46464 (local address 127.9.93.62:38095)
0928 16:49:39.488810 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:39.488817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:39.488988 (+ 171us) server_negotiation.cc:407] Connection header received
0928 16:49:39.489161 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:39.489167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:39.489258 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:39.489374 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:39.490740 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.492018 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.493424 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.494134 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.494290 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:39.496908 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:39.496916 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:39.496921 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:39.496977 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:39.500258 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:39.501063 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:39.501068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:39.501070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:39.501231 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:39.501626 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:39.501630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:39.501632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:39.501967 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:39.502245 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:39.502594 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 16:49:39.502867 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 16:49:39.503897 10719 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46464
I20260928 16:49:39.505959 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 10790
I20260928 16:49:39.523777 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:39.506994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.9.93.1:44601)
0928 16:49:39.507221 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:39.507227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:39.508713 (+ 1486us) server_negotiation.cc:407] Connection header received
0928 16:49:39.508858 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:39.508864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:39.508963 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:39.509068 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:39.510442 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.511790 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.513265 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.513945 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.514106 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:39.516865 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:39.516871 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:39.516874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:39.516922 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:39.520376 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:39.521263 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:39.521270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:39.521275 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:39.521490 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:39.521883 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:39.521888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:39.521891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:39.522328 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:39.522621 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:39.523001 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 16:49:39.523331 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260928 16:49:39.524736 10921 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38230
W20260928 16:49:39.525399 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 10790: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:49:39.525609 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 10790
I20260928 16:49:39.542148 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 10930
Sep 28 16:49:39 dist-test-slave-wzcb krb5kdc[10674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614178, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.93.2@KRBTEST.COM
I20260928 16:49:39.560984 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:39.543060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59632 (local address 127.9.93.2:33345)
0928 16:49:39.543336 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:39.543344 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:39.543368 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:49:39.543508 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:39.543514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:39.543597 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:39.543688 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:39.545005 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.545875 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.547972 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.548666 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.548826 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:39.554001 (+ 5175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:39.554022 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:39.554041 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:39.554100 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:39.557452 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:39.558464 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:39.558473 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:39.558475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:39.558681 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:39.559155 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:39.559158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:39.559160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:39.559708 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:39.560016 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:39.560362 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 16:49:39.560647 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260928 16:49:39.561738 11062 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59632
W20260928 16:49:39.562469 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 10930: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:49:39.562685 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 10930
I20260928 16:49:39.589732 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 11071
Sep 28 16:49:39 dist-test-slave-wzcb krb5kdc[10674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614178, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.93.3@KRBTEST.COM
I20260928 16:49:39.609448 11288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:39.590738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47952 (local address 127.9.93.3:33469)
0928 16:49:39.591723 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:39.591728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:39.591748 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:49:39.591810 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:39.591814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:39.591905 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:39.592035 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:39.594021 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.594800 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.597187 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:39.597757 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:39.597935 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:39.602689 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:39.602708 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:39.602727 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:39.602784 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:39.606057 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:39.606927 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:39.606937 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:39.606940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:39.607112 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:39.607611 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:39.607616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:39.607618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:39.608056 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:39.608383 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:39.608775 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 16:49:39.609058 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":858,"thread_start_us":638,"threads_started":1}
W20260928 16:49:39.610320 11202 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47952
W20260928 16:49:39.611070 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 11071: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:49:39.611248 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 11071
I20260928 16:49:39.626338 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 10687
I20260928 16:49:39.627702 10759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:49:39.751185 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 10687
2026-09-28T16:49:39Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4896 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:49:39 dist-test-slave-wzcb krb5kdc[11292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:49:39 dist-test-slave-wzcb krb5kdc[11292](info): set up 2 sockets
Sep 28 16:49:39 dist-test-slave-wzcb krb5kdc[11292](info): commencing operation
krb5kdc: starting...
W20260928 16:49:40.762063 9588 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:49:40 dist-test-slave-wzcb krb5kdc[11292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:49:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:49:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:49:40.975700 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:45277
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:45479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:45277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:41.341051 11305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:41.341394 11305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:41.341524 11305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:41.349283 11305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:41.349422 11305 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 16:49:41.349470 11305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:41.349560 11305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:41.349650 11305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:41.349745 11305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:41.360375 11305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:45479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:45277
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:45277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:41.361934 11305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:41.363586 11305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:41.377413 11313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:41.377517 11310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:41.377395 11311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:41.378490 11305 server_base.cc:1034] running on GCE node
I20260928 16:49:41.379501 11305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:41.381330 11305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:41.382663 11305 hybrid_clock.cc:648] HybridClock initialized: now 1790614181382570 us; error 59 us; skew 500 ppm
I20260928 16:49:41.383076 11305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:41 dist-test-slave-wzcb krb5kdc[11292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614181, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:41.386700 11305 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:49:41.388520 11305 webserver.cc:460] Webserver started at http://127.9.93.62:46835/ using document root <none> and password file <none>
I20260928 16:49:41.389360 11305 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:41.389479 11305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:41.389832 11305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:41.393517 11305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "facb04e3edf241c2a7d0ceb159d708b6"
format_stamp: "Formatted at 2026-09-28 16:49:41 on dist-test-slave-wzcb"
server_key: "1f166edc6554fe6d4a6bf2fad07438e0"
server_key_iv: "1681d8f2d8c2d9d4282aa6b49f6f3757"
server_key_version: "encryptionkey@0"
I20260928 16:49:41.394462 11305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "facb04e3edf241c2a7d0ceb159d708b6"
format_stamp: "Formatted at 2026-09-28 16:49:41 on dist-test-slave-wzcb"
server_key: "1f166edc6554fe6d4a6bf2fad07438e0"
server_key_iv: "1681d8f2d8c2d9d4282aa6b49f6f3757"
server_key_version: "encryptionkey@0"
I20260928 16:49:41.400786 11305 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 16:49:41.404930 11320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:41.406723 11305 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:49:41.406947 11305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "facb04e3edf241c2a7d0ceb159d708b6"
format_stamp: "Formatted at 2026-09-28 16:49:41 on dist-test-slave-wzcb"
server_key: "1f166edc6554fe6d4a6bf2fad07438e0"
server_key_iv: "1681d8f2d8c2d9d4282aa6b49f6f3757"
server_key_version: "encryptionkey@0"
I20260928 16:49:41.407208 11305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:41.436276 11305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:41.441182 11305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:41.441533 11305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:41.463663 11305 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:45277
I20260928 16:49:41.463687 11381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:45277 every 8 connection(s)
I20260928 16:49:41.465277 11305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:49:41.469918 11382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:41.472610 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 11305
I20260928 16:49:41.472870 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:49:41.473793 9588 external_mini_cluster.cc:1428] Setting key 353c44f64f7ed4476041d8d0fa5e12ca
I20260928 16:49:41.484258 11382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6: Bootstrap starting.
I20260928 16:49:41.489176 11382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:41.490749 11382 log.cc:824] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:49:41 dist-test-slave-wzcb krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:49:41.494858 11382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6: No bootstrap required, opened a new log
I20260928 16:49:41.501173 11385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:41.476331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41056 (local address 127.9.93.62:45277)
0928 16:49:41.477403 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:41.477419 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:41.477472 (+ 53us) server_negotiation.cc:407] Connection header received
0928 16:49:41.479449 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:41.479481 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:41.479992 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:41.480438 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:41.482418 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:41.483939 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:41.485824 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:41.486679 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:41.486841 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:41.492337 (+ 5496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:41.492361 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:41.492420 (+ 59us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:41.492524 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:41.495951 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:41.497108 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:41.497117 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:41.497125 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:41.497346 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:41.497846 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:41.497851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:41.497853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:41.498549 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:41.498800 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:41.499385 (+ 585us) server_negotiation.cc:299] Negotiation successful
0928 16:49:41.499829 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":461,"threads_started":1}
I20260928 16:49:41.504670 11382 raft_consensus.cc:348] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "facb04e3edf241c2a7d0ceb159d708b6" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45277 } }
I20260928 16:49:41.505132 11382 raft_consensus.cc:374] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:41.505245 11382 raft_consensus.cc:729] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: facb04e3edf241c2a7d0ceb159d708b6, State: Initialized, Role: FOLLOWER
I20260928 16:49:41.505928 11382 consensus_queue.cc:260] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "facb04e3edf241c2a7d0ceb159d708b6" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45277 } }
I20260928 16:49:41.506220 11382 raft_consensus.cc:388] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:49:41.506374 11382 raft_consensus.cc:482] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:49:41.506577 11382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:41.509358 11382 raft_consensus.cc:504] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "facb04e3edf241c2a7d0ceb159d708b6" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45277 } }
I20260928 16:49:41.510064 11382 leader_election.cc:302] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: facb04e3edf241c2a7d0ceb159d708b6; no voters:
I20260928 16:49:41.510778 11382 leader_election.cc:288] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:49:41.510937 11387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:41.512310 11387 raft_consensus.cc:686] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6 [term 1 LEADER]: Becoming Leader. State: Replica: facb04e3edf241c2a7d0ceb159d708b6, State: Running, Role: LEADER
I20260928 16:49:41.513047 11387 consensus_queue.cc:237] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "facb04e3edf241c2a7d0ceb159d708b6" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45277 } }
I20260928 16:49:41.513830 11382 sys_catalog.cc:564] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:41.516189 11389 sys_catalog.cc:455] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader facb04e3edf241c2a7d0ceb159d708b6. Latest consensus state: current_term: 1 leader_uuid: "facb04e3edf241c2a7d0ceb159d708b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "facb04e3edf241c2a7d0ceb159d708b6" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45277 } } }
I20260928 16:49:41.516687 11389 sys_catalog.cc:458] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:41.516165 11388 sys_catalog.cc:455] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "facb04e3edf241c2a7d0ceb159d708b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "facb04e3edf241c2a7d0ceb159d708b6" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45277 } } }
I20260928 16:49:41.516855 11388 sys_catalog.cc:458] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:41.517760 11394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:49:41.523204 11394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:49:41.534085 11394 catalog_manager.cc:1347] Generated new cluster ID: 8b579302776d46dc9df0111a2e6ea1ab
I20260928 16:49:41.534236 11394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:49:41.549458 11394 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:49:41.550987 11394 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:49:41.559122 11394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6: Generated new TSK 0
I20260928 16:49:41.560168 11394 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
I20260928 16:49:41.627897 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:45277
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:45479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:42.031487 11408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:42.031840 11408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:42.031972 11408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:42.040364 11408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:42.040529 11408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:42.040647 11408 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 16:49:42.040711 11408 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:49:42.040793 11408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:49:42.052729 11408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:45479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.93.62:45277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:42.054239 11408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:42.055917 11408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:42.071187 11416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:42.071272 11414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:42.071282 11413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:42.072803 11408 server_base.cc:1034] running on GCE node
I20260928 16:49:42.073626 11408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:42.074848 11408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:42.076125 11408 hybrid_clock.cc:648] HybridClock initialized: now 1790614182076037 us; error 79 us; skew 500 ppm
I20260928 16:49:42.076561 11408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:42 dist-test-slave-wzcb krb5kdc[11292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614182, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:42.080590 11408 init.cc:377] Logged in from keytab as kudu/127.9.93.1@KRBTEST.COM (short username kudu)
I20260928 16:49:42.082592 11408 webserver.cc:460] Webserver started at http://127.9.93.1:33675/ using document root <none> and password file <none>
I20260928 16:49:42.083544 11408 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:42.083709 11408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:42.084067 11408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:42.087749 11408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "d9567e6c8c4048d38639fbc991c3e211"
format_stamp: "Formatted at 2026-09-28 16:49:42 on dist-test-slave-wzcb"
server_key: "9663a2bc1734470e3a878beae0d4b76a"
server_key_iv: "a7c24955115eb1d97215494eb81893cf"
server_key_version: "encryptionkey@0"
I20260928 16:49:42.088723 11408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "d9567e6c8c4048d38639fbc991c3e211"
format_stamp: "Formatted at 2026-09-28 16:49:42 on dist-test-slave-wzcb"
server_key: "9663a2bc1734470e3a878beae0d4b76a"
server_key_iv: "a7c24955115eb1d97215494eb81893cf"
server_key_version: "encryptionkey@0"
I20260928 16:49:42.095235 11408 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260928 16:49:42.099977 11423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:42.101780 11408 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:49:42.101999 11408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "d9567e6c8c4048d38639fbc991c3e211"
format_stamp: "Formatted at 2026-09-28 16:49:42 on dist-test-slave-wzcb"
server_key: "9663a2bc1734470e3a878beae0d4b76a"
server_key_iv: "a7c24955115eb1d97215494eb81893cf"
server_key_version: "encryptionkey@0"
I20260928 16:49:42.102255 11408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:42.126560 11408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:42.131867 11408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:42.132218 11408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:42.135464 11408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:49:42.135624 11408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:42.135766 11408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:49:42.135880 11408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:49:42 dist-test-slave-wzcb krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614182, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:49:42.161823 11432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:42.136028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:44251 (local address 127.9.93.62:45277)
0928 16:49:42.139788 (+ 3760us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:42.139795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:42.139824 (+ 29us) server_negotiation.cc:407] Connection header received
0928 16:49:42.141134 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:42.141140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:42.141258 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:42.141365 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:42.143323 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:42.144365 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:42.147268 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:42.147851 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:42.148019 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:42.153900 (+ 5881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:42.153907 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:42.153911 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:42.153953 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:42.157262 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:42.158671 (+ 1409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:42.158677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:42.158679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:42.158829 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:42.159662 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:42.159669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:42.159673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:42.159999 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:42.160138 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:42.161310 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0928 16:49:42.161525 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3614,"thread_start_us":3463,"threads_started":1}
I20260928 16:49:42.164255 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:42.136916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:45277 (local address 127.9.93.1:44251)
0928 16:49:42.138102 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:42.138178 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:42.140769 (+ 2591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:42.141672 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:42.141682 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:42.142290 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:42.143084 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:42.143111 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:42.144559 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:42.144563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:42.146968 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:42.146988 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:42.148263 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:42.148277 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:42.148712 (+ 435us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:42.149659 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:42.149686 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:42.153520 (+ 3834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:42.157469 (+ 3949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:42.157482 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:42.157517 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:42.158426 (+ 909us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:42.159002 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:42.159009 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:42.159014 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:42.159484 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:42.160155 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:42.160166 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:42.160829 (+ 663us) client_negotiation.cc:769] Sending connection context
0928 16:49:42.161211 (+ 382us) client_negotiation.cc:240] Negotiation successful
0928 16:49:42.162000 (+ 789us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":471,"threads_started":1}
I20260928 16:49:42.192523 11408 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:38335
I20260928 16:49:42.192611 11544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:38335 every 8 connection(s)
I20260928 16:49:42.194247 11408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:49:42.194275 11432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:42.172961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:57675 (local address 127.9.93.62:45277)
0928 16:49:42.173374 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:42.173381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:42.173971 (+ 590us) server_negotiation.cc:407] Connection header received
0928 16:49:42.174259 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:42.174265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:42.174368 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:42.174499 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:42.176411 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:42.177360 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:42.179814 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:42.180323 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:42.180468 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:42.187547 (+ 7079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:42.187556 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:42.187561 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:42.187616 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:42.190358 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:42.191589 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:42.191597 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:42.191601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:42.191824 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:42.192380 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:42.192386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:42.192389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:42.192916 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:42.193107 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:42.193599 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 16:49:42.193894 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233}
I20260928 16:49:42.194360 11502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:42.172869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:45277 (local address 127.9.93.1:57675)
0928 16:49:42.173807 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:42.173838 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:42.174074 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:42.174569 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:42.174575 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:42.175475 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:42.176137 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:42.176162 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:42.177552 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:42.177559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:42.179593 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:42.179624 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:42.180660 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:42.180671 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:42.180901 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:42.185758 (+ 4857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:42.185798 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:42.187175 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:42.190560 (+ 3385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:42.190567 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:42.190573 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:42.191355 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:42.191987 (+ 632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:42.191993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:42.191996 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:42.192201 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:42.193053 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:42.193058 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:42.193295 (+ 237us) client_negotiation.cc:769] Sending connection context
0928 16:49:42.193438 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 16:49:42.193861 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":631,"threads_started":1}
I20260928 16:49:42.196977 11545 heartbeater.cc:347] Connected to a master server at 127.9.93.62:45277
I20260928 16:49:42.197263 11545 heartbeater.cc:464] Registering TS with master...
I20260928 16:49:42.198199 11545 heartbeater.cc:511] Master 127.9.93.62:45277 requested a full tablet report, sending...
I20260928 16:49:42.200639 11337 ts_manager.cc:194] Registered new tserver with Master: d9567e6c8c4048d38639fbc991c3e211 (127.9.93.1:38335)
I20260928 16:49:42.202333 11337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.1@KRBTEST.COM'} at 127.9.93.1:44251
I20260928 16:49:42.204461 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 11408
I20260928 16:49:42.204677 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:49:42.205325 9588 external_mini_cluster.cc:1428] Setting key bc4988963d1e6d2410ada1c0cafe9d40
WARNING: no policy specified for kudu/127.9.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
I20260928 16:49:42.252154 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:45277
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:45479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:42.622808 11549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:42.623162 11549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:42.623333 11549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:42.631462 11549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:42.631618 11549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:42.631764 11549 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 16:49:42.631835 11549 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:49:42.631953 11549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:49:42.643664 11549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:45479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.93.62:45277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:42.645663 11549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:42.647519 11549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:42.663175 11554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:42.663115 11555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:42.663138 11557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:42.664306 11549 server_base.cc:1034] running on GCE node
I20260928 16:49:42.665124 11549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:42.666292 11549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:42.667512 11549 hybrid_clock.cc:648] HybridClock initialized: now 1790614182667437 us; error 62 us; skew 500 ppm
I20260928 16:49:42.667930 11549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:42 dist-test-slave-wzcb krb5kdc[11292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614182, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:42.671545 11549 init.cc:377] Logged in from keytab as kudu/127.9.93.2@KRBTEST.COM (short username kudu)
I20260928 16:49:42.673456 11549 webserver.cc:460] Webserver started at http://127.9.93.2:45791/ using document root <none> and password file <none>
I20260928 16:49:42.674306 11549 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:42.674437 11549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:42.674860 11549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:42.678436 11549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182"
format_stamp: "Formatted at 2026-09-28 16:49:42 on dist-test-slave-wzcb"
server_key: "23444c1813f00c9677cc6317fd77c02e"
server_key_iv: "1af6e45db8136235b26f54dbd4adc5fc"
server_key_version: "encryptionkey@0"
I20260928 16:49:42.679433 11549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182"
format_stamp: "Formatted at 2026-09-28 16:49:42 on dist-test-slave-wzcb"
server_key: "23444c1813f00c9677cc6317fd77c02e"
server_key_iv: "1af6e45db8136235b26f54dbd4adc5fc"
server_key_version: "encryptionkey@0"
I20260928 16:49:42.685829 11549 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:49:42.690320 11564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:42.692126 11549 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 16:49:42.692360 11549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182"
format_stamp: "Formatted at 2026-09-28 16:49:42 on dist-test-slave-wzcb"
server_key: "23444c1813f00c9677cc6317fd77c02e"
server_key_iv: "1af6e45db8136235b26f54dbd4adc5fc"
server_key_version: "encryptionkey@0"
I20260928 16:49:42.692673 11549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:42.714218 11549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:42.719658 11549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:42.719995 11549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:42.723474 11549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:49:42.723616 11549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:42.723752 11549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:49:42.723842 11549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:49:42 dist-test-slave-wzcb krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614182, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:49:42.754456 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:42.724317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:49383 (local address 127.9.93.62:45277)
0928 16:49:42.724898 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:42.724904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:42.726349 (+ 1445us) server_negotiation.cc:407] Connection header received
0928 16:49:42.729712 (+ 3363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:42.729717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:42.729825 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:42.729980 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:42.733344 (+ 3364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:42.734208 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:42.737701 (+ 3493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:42.738498 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:42.738732 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:42.745229 (+ 6497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:42.745239 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:42.745245 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:42.745312 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:42.749498 (+ 4186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:42.750627 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:42.750634 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:42.750639 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:42.750868 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:42.751697 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:42.751702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:42.751704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:42.752186 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:42.752368 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:42.753714 (+ 1346us) server_negotiation.cc:299] Negotiation successful
0928 16:49:42.754031 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":289,"threads_started":1}
I20260928 16:49:42.756327 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:42.725057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:45277 (local address 127.9.93.2:49383)
0928 16:49:42.726088 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:42.726161 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:42.729225 (+ 3064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:42.730372 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:42.730387 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:42.731581 (+ 1194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:42.733021 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:42.733070 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:42.734428 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:42.734434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:42.737337 (+ 2903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:42.737369 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:42.738913 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:42.738922 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:42.739399 (+ 477us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:42.740662 (+ 1263us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:42.740709 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:42.744729 (+ 4020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:42.749755 (+ 5026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:42.749766 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:42.749784 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:42.750439 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:42.751046 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:42.751052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:42.751054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:42.751503 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:42.752373 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:42.752384 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:42.753118 (+ 734us) client_negotiation.cc:769] Sending connection context
0928 16:49:42.753584 (+ 466us) client_negotiation.cc:240] Negotiation successful
0928 16:49:42.754322 (+ 738us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":356,"threads_started":1}
I20260928 16:49:42.782099 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:42.763552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:58561 (local address 127.9.93.62:45277)
0928 16:49:42.763754 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:42.763761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:42.764543 (+ 782us) server_negotiation.cc:407] Connection header received
0928 16:49:42.764794 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:42.764798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:42.764872 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:42.764987 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:42.766814 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:42.767875 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:42.769479 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:42.770311 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:42.770483 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:42.773624 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:42.773634 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:42.773640 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:42.773699 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:42.777482 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:42.778855 (+ 1373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:42.778863 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:42.778867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:42.779086 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:42.779719 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:42.779726 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:42.779730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:42.780333 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:42.780517 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:42.781344 (+ 827us) server_negotiation.cc:299] Negotiation successful
0928 16:49:42.781676 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:49:42.782262 11643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:42.763571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:45277 (local address 127.9.93.2:58561)
0928 16:49:42.764405 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:42.764424 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:42.764613 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:42.765078 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:42.765088 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:42.765932 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:42.766580 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:42.766606 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:42.768077 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:42.768081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:42.769235 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:42.769257 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:42.770664 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:42.770670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:42.770850 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:42.771688 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:42.771718 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:42.773218 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:42.777730 (+ 4512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:42.777739 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:42.777745 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:42.778635 (+ 890us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:42.779264 (+ 629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:42.779272 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:42.779277 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:42.779554 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:42.780613 (+ 1059us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:42.780623 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:42.780966 (+ 343us) client_negotiation.cc:769] Sending connection context
0928 16:49:42.781195 (+ 229us) client_negotiation.cc:240] Negotiation successful
0928 16:49:42.781688 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":547,"threads_started":1}
I20260928 16:49:42.789420 11549 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:41673
I20260928 16:49:42.789465 11684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:41673 every 8 connection(s)
I20260928 16:49:42.790958 11549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:49:42.793488 11685 heartbeater.cc:347] Connected to a master server at 127.9.93.62:45277
I20260928 16:49:42.793797 11685 heartbeater.cc:464] Registering TS with master...
I20260928 16:49:42.794493 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 11549
I20260928 16:49:42.794692 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:49:42.794929 11685 heartbeater.cc:511] Master 127.9.93.62:45277 requested a full tablet report, sending...
I20260928 16:49:42.795382 9588 external_mini_cluster.cc:1428] Setting key 096e663239da26bc5de6493dd75dea04
I20260928 16:49:42.797050 11337 ts_manager.cc:194] Registered new tserver with Master: ed4b3cc6d19f4a6d9d4b126b8ed6d182 (127.9.93.2:41673)
I20260928 16:49:42.798231 11337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.2@KRBTEST.COM'} at 127.9.93.2:49383
WARNING: no policy specified for kudu/127.9.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
I20260928 16:49:42.844116 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:45277
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:45479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:49:43.205997 11545 heartbeater.cc:503] Master 127.9.93.62:45277 was elected leader, sending a full tablet report...
W20260928 16:49:43.223275 11689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:43.223708 11689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:43.223873 11689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:43.232048 11689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:43.232223 11689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:43.232360 11689 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 16:49:43.232432 11689 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:49:43.232656 11689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:49:43.243763 11689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:45479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.93.62:45277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:43.245393 11689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:43.247004 11689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:43.261902 11694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:43.261831 11697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:43.261869 11695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:43.262629 11689 server_base.cc:1034] running on GCE node
I20260928 16:49:43.263381 11689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:43.264596 11689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:43.265913 11689 hybrid_clock.cc:648] HybridClock initialized: now 1790614183265846 us; error 71 us; skew 500 ppm
I20260928 16:49:43.266299 11689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:43 dist-test-slave-wzcb krb5kdc[11292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614183, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:43.270109 11689 init.cc:377] Logged in from keytab as kudu/127.9.93.3@KRBTEST.COM (short username kudu)
I20260928 16:49:43.271970 11689 webserver.cc:460] Webserver started at http://127.9.93.3:38999/ using document root <none> and password file <none>
I20260928 16:49:43.272771 11689 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:43.272894 11689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:43.273203 11689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:43.276863 11689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "f8e2fe45943745edb3479645dd18575c"
format_stamp: "Formatted at 2026-09-28 16:49:43 on dist-test-slave-wzcb"
server_key: "6d532ce8e34b04a8b5fd33fb7544a5ed"
server_key_iv: "359d8766d3659568fb4ce2319b5dfe96"
server_key_version: "encryptionkey@0"
I20260928 16:49:43.277776 11689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "f8e2fe45943745edb3479645dd18575c"
format_stamp: "Formatted at 2026-09-28 16:49:43 on dist-test-slave-wzcb"
server_key: "6d532ce8e34b04a8b5fd33fb7544a5ed"
server_key_iv: "359d8766d3659568fb4ce2319b5dfe96"
server_key_version: "encryptionkey@0"
I20260928 16:49:43.284674 11689 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 16:49:43.288951 11704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:43.290644 11689 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:49:43.290827 11689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "f8e2fe45943745edb3479645dd18575c"
format_stamp: "Formatted at 2026-09-28 16:49:43 on dist-test-slave-wzcb"
server_key: "6d532ce8e34b04a8b5fd33fb7544a5ed"
server_key_iv: "359d8766d3659568fb4ce2319b5dfe96"
server_key_version: "encryptionkey@0"
I20260928 16:49:43.291050 11689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:43.308769 11689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:43.313723 11689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:43.314088 11689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:43.317459 11689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:49:43.317662 11689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 16:49:43.317855 11689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:49:43.317976 11689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:49:43 dist-test-slave-wzcb krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614183, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:49:43.346580 11714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:43.318739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:45277 (local address 127.9.93.3:44909)
0928 16:49:43.319723 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:43.319767 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:43.321854 (+ 2087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:43.322885 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:43.322897 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:43.323927 (+ 1030us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:43.325131 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:43.325181 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.326750 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.326757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:43.329562 (+ 2805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:43.329593 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.330959 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.330969 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:43.331344 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:43.332304 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:43.332347 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:43.335989 (+ 3642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:43.340457 (+ 4468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:43.340474 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:43.340504 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:43.341379 (+ 875us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:43.341885 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:43.341894 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:43.341897 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:43.342288 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:43.343046 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:43.343060 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:43.343601 (+ 541us) client_negotiation.cc:769] Sending connection context
0928 16:49:43.344022 (+ 421us) client_negotiation.cc:240] Negotiation successful
0928 16:49:43.344720 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":379,"threads_started":1}
I20260928 16:49:43.347792 11712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:43.317875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:44909 (local address 127.9.93.62:45277)
0928 16:49:43.318352 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:43.318359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:43.319917 (+ 1558us) server_negotiation.cc:407] Connection header received
0928 16:49:43.322251 (+ 2334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:43.322257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:43.322371 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:43.322498 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:43.325412 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.326534 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.329840 (+ 3306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.330436 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.330582 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:43.336360 (+ 5778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:43.336366 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:43.336370 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:43.336429 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:43.340183 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:43.341586 (+ 1403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:43.341591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:43.341593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:43.341739 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:43.342445 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:43.342451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:43.342455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:43.342875 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:43.343042 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:43.347070 (+ 4028us) server_negotiation.cc:299] Negotiation successful
0928 16:49:43.347411 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":238,"threads_started":1}
I20260928 16:49:43.369237 11689 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:41059
I20260928 16:49:43.369343 11824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:41059 every 8 connection(s)
I20260928 16:49:43.371238 11689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:49:43.374277 11825 heartbeater.cc:347] Connected to a master server at 127.9.93.62:45277
I20260928 16:49:43.374624 11825 heartbeater.cc:464] Registering TS with master...
I20260928 16:49:43.375519 11825 heartbeater.cc:511] Master 127.9.93.62:45277 requested a full tablet report, sending...
I20260928 16:49:43.377250 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 11689
I20260928 16:49:43.377453 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:49:43.377413 11337 ts_manager.cc:194] Registered new tserver with Master: f8e2fe45943745edb3479645dd18575c (127.9.93.3:41059)
I20260928 16:49:43.378037 9588 external_mini_cluster.cc:1428] Setting key 477906c2c9612e829fd719d15f6e8fc7
I20260928 16:49:43.378499 11337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.3@KRBTEST.COM'} at 127.9.93.3:44909
I20260928 16:49:43.380612 11808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:43.355088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:45277 (local address 127.9.93.3:60733)
0928 16:49:43.363649 (+ 8561us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:43.363688 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:43.363975 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:43.364522 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:43.364527 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:43.365365 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:43.366040 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:43.366062 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.367780 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.367786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:43.369657 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:43.369683 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.370921 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.370928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:43.371103 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:43.371925 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:43.371964 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:43.373437 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:43.377303 (+ 3866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:43.377309 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:43.377313 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:43.377898 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:43.378393 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:43.378398 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:43.378403 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:43.378583 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:43.379247 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:43.379252 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:43.379516 (+ 264us) client_negotiation.cc:769] Sending connection context
0928 16:49:43.379685 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 16:49:43.380091 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4275,"thread_start_us":4084,"threads_started":1}
I20260928 16:49:43.380621 11712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:43.355775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:60733 (local address 127.9.93.62:45277)
0928 16:49:43.355999 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:43.356008 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:43.363861 (+ 7853us) server_negotiation.cc:407] Connection header received
0928 16:49:43.364169 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:43.364181 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:43.364305 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:43.364468 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:43.366353 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.367579 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.369910 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.370615 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.370790 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:43.373753 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:43.373761 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:43.373766 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:43.373841 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:43.377122 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:43.378052 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:43.378059 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:43.378063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:43.378255 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:43.378731 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:43.378736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:43.378739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:43.379120 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:43.379261 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:43.379824 (+ 563us) server_negotiation.cc:299] Negotiation successful
0928 16:49:43.380255 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 16:49:43.381620 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 16:49:43 dist-test-slave-wzcb krb5kdc[11292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 16:49:43 dist-test-slave-wzcb krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:49:43.434122 11712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:43.419085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53032 (local address 127.9.93.62:45277)
0928 16:49:43.419355 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:43.419363 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:43.419706 (+ 343us) server_negotiation.cc:407] Connection header received
0928 16:49:43.419945 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:43.419952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:43.420041 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:43.420143 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:43.421653 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.422540 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.423974 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.424415 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.424538 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:43.428718 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:43.428723 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:43.428727 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:43.428774 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:43.431499 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:43.432210 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:43.432215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:43.432217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:43.432377 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:43.432756 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:43.432759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:43.432761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:43.433091 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:43.433295 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:43.433579 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 16:49:43.433769 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:49:43.438737 11337 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:49:43.446877 11337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:49:43.477470 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:43.461767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48528 (local address 127.9.93.1:38335)
0928 16:49:43.462794 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:43.462803 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:43.462834 (+ 31us) server_negotiation.cc:407] Connection header received
0928 16:49:43.462926 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:43.462932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:43.463209 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:43.463513 (+ 304us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:49:43.465617 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.467246 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.472261 (+ 5015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.474156 (+ 1895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.474281 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:43.474556 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:43.476893 (+ 2337us) server_negotiation.cc:299] Negotiation successful
0928 16:49:43.477138 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":854,"thread_start_us":676,"threads_started":1}
I20260928 16:49:43.478382 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:43.461557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:38335 (local address 127.0.0.1:48528)
0928 16:49:43.462449 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:43.462507 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:43.462701 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:43.463799 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:43.463809 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:43.463851 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:49:43.464248 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:43.464260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.467513 (+ 3253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.467518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:43.472020 (+ 4502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:43.472040 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.476243 (+ 4203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.476249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:43.476563 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:43.476636 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 16:49:43.477736 (+ 1100us) client_negotiation.cc:240] Negotiation successful
0928 16:49:43.478009 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":445,"threads_started":1}
I20260928 16:49:43.483641 11840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:43.462954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:41059 (local address 127.0.0.1:48150)
0928 16:49:43.470644 (+ 7690us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:43.470669 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:43.470839 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:43.472296 (+ 1457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:43.472303 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:43.472339 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:49:43.472938 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:43.472954 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.475640 (+ 2686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.475648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:43.479256 (+ 3608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:43.479270 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.482420 (+ 3150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.482430 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:43.482720 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:43.482768 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 16:49:43.482888 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 16:49:43.483100 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7494,"thread_start_us":7326,"threads_started":1}
I20260928 16:49:43.485014 11841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:43.463494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48150 (local address 127.9.93.3:41059)
0928 16:49:43.469968 (+ 6474us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:43.469977 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:43.471011 (+ 1034us) server_negotiation.cc:407] Connection header received
0928 16:49:43.471153 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:43.471164 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:43.471604 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:43.471871 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:49:43.473256 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.475388 (+ 2132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.479563 (+ 4175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.481636 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.481815 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:43.482195 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:43.484396 (+ 2201us) server_negotiation.cc:299] Negotiation successful
0928 16:49:43.484650 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6246,"thread_start_us":6006,"threads_started":1}
I20260928 16:49:43.490893 11755 tablet_service.cc:1467] Processing CreateTablet for tablet f53ded63cf5c4e29b1464019f190e126 (DEFAULT_TABLE table=test-table [id=09fc40cce8394521a5002861136883cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:49:43.491518 11475 tablet_service.cc:1467] Processing CreateTablet for tablet f53ded63cf5c4e29b1464019f190e126 (DEFAULT_TABLE table=test-table [id=09fc40cce8394521a5002861136883cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:49:43.492823 11475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53ded63cf5c4e29b1464019f190e126. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:43.492858 11755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53ded63cf5c4e29b1464019f190e126. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:43.493129 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:43.471059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:41673 (local address 127.0.0.1:60960)
0928 16:49:43.478788 (+ 7729us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:43.478815 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:43.478961 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:43.485560 (+ 6599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:43.485566 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:43.485604 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:49:43.485839 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:43.485856 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.488327 (+ 2471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.488333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:43.490240 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:43.490261 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.492229 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.492236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:43.492439 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:43.492479 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 16:49:43.492555 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 16:49:43.492714 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7591,"thread_start_us":14306,"threads_started":1}
I20260928 16:49:43.494356 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:43.471104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60960 (local address 127.9.93.2:41673)
0928 16:49:43.484449 (+ 13345us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:43.484464 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:43.484511 (+ 47us) server_negotiation.cc:407] Connection header received
0928 16:49:43.484668 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:43.484681 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:43.485098 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:43.486097 (+ 999us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:49:43.486419 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.488098 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.490480 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.492015 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.493115 (+ 1100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:43.493735 (+ 620us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:43.493822 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 16:49:43.494040 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13071,"thread_start_us":12861,"threads_started":1}
I20260928 16:49:43.498234 11615 tablet_service.cc:1467] Processing CreateTablet for tablet f53ded63cf5c4e29b1464019f190e126 (DEFAULT_TABLE table=test-table [id=09fc40cce8394521a5002861136883cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:49:43.499941 11615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53ded63cf5c4e29b1464019f190e126. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:43.506050 11844 tablet_bootstrap.cc:492] T f53ded63cf5c4e29b1464019f190e126 P d9567e6c8c4048d38639fbc991c3e211: Bootstrap starting.
I20260928 16:49:43.509773 11844 tablet_bootstrap.cc:654] T f53ded63cf5c4e29b1464019f190e126 P d9567e6c8c4048d38639fbc991c3e211: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:43.511029 11844 log.cc:824] T f53ded63cf5c4e29b1464019f190e126 P d9567e6c8c4048d38639fbc991c3e211: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:43.513312 11846 tablet_bootstrap.cc:492] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c: Bootstrap starting.
I20260928 16:49:43.515334 11844 tablet_bootstrap.cc:492] T f53ded63cf5c4e29b1464019f190e126 P d9567e6c8c4048d38639fbc991c3e211: No bootstrap required, opened a new log
I20260928 16:49:43.515684 11844 ts_tablet_manager.cc:1397] T f53ded63cf5c4e29b1464019f190e126 P d9567e6c8c4048d38639fbc991c3e211: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260928 16:49:43.517211 11847 tablet_bootstrap.cc:492] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Bootstrap starting.
I20260928 16:49:43.518213 11846 tablet_bootstrap.cc:654] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:43.519898 11846 log.cc:824] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:43.520607 11847 tablet_bootstrap.cc:654] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:43.521956 11847 log.cc:824] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:43.523728 11846 tablet_bootstrap.cc:492] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c: No bootstrap required, opened a new log
I20260928 16:49:43.524050 11846 ts_tablet_manager.cc:1397] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260928 16:49:43.524710 11847 tablet_bootstrap.cc:492] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182: No bootstrap required, opened a new log
I20260928 16:49:43.524966 11847 ts_tablet_manager.cc:1397] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 16:49:43.527549 11844 raft_consensus.cc:348] T f53ded63cf5c4e29b1464019f190e126 P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } }
I20260928 16:49:43.528071 11844 raft_consensus.cc:374] T f53ded63cf5c4e29b1464019f190e126 P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:43.528182 11844 raft_consensus.cc:729] T f53ded63cf5c4e29b1464019f190e126 P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9567e6c8c4048d38639fbc991c3e211, State: Initialized, Role: FOLLOWER
I20260928 16:49:43.528927 11844 consensus_queue.cc:260] T f53ded63cf5c4e29b1464019f190e126 P d9567e6c8c4048d38639fbc991c3e211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } }
I20260928 16:49:43.531389 11844 ts_tablet_manager.cc:1428] T f53ded63cf5c4e29b1464019f190e126 P d9567e6c8c4048d38639fbc991c3e211: Time spent starting tablet: real 0.016s user 0.009s sys 0.006s
I20260928 16:49:43.534519 11847 raft_consensus.cc:348] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } }
I20260928 16:49:43.534962 11847 raft_consensus.cc:374] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:43.535080 11847 raft_consensus.cc:729] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed4b3cc6d19f4a6d9d4b126b8ed6d182, State: Initialized, Role: FOLLOWER
I20260928 16:49:43.534862 11846 raft_consensus.cc:348] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } }
I20260928 16:49:43.535449 11846 raft_consensus.cc:374] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:43.535557 11846 raft_consensus.cc:729] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e2fe45943745edb3479645dd18575c, State: Initialized, Role: FOLLOWER
I20260928 16:49:43.535924 11847 consensus_queue.cc:260] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } }
I20260928 16:49:43.536146 11846 consensus_queue.cc:260] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } }
I20260928 16:49:43.537345 11825 heartbeater.cc:503] Master 127.9.93.62:45277 was elected leader, sending a full tablet report...
I20260928 16:49:43.538225 11846 ts_tablet_manager.cc:1428] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260928 16:49:43.539080 11685 heartbeater.cc:503] Master 127.9.93.62:45277 was elected leader, sending a full tablet report...
I20260928 16:49:43.539904 11847 ts_tablet_manager.cc:1428] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260928 16:49:43.543292 11686 tablet.cc:2369] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:49:43.624302 11826 tablet.cc:2369] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:49:43.697733 11546 tablet.cc:2369] T f53ded63cf5c4e29b1464019f190e126 P d9567e6c8c4048d38639fbc991c3e211: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:49:43.853157 11851 raft_consensus.cc:482] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:49:43.853560 11851 raft_consensus.cc:504] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } }
I20260928 16:49:43.855566 11851 leader_election.cc:288] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9567e6c8c4048d38639fbc991c3e211 (127.9.93.1:38335), f8e2fe45943745edb3479645dd18575c (127.9.93.3:41059)
I20260928 16:49:43.856310 11852 raft_consensus.cc:482] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:49:43.856638 11852 raft_consensus.cc:504] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } }
I20260928 16:49:43.860092 11852 leader_election.cc:288] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9567e6c8c4048d38639fbc991c3e211 (127.9.93.1:38335), ed4b3cc6d19f4a6d9d4b126b8ed6d182 (127.9.93.2:41673)
I20260928 16:49:43.865675 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:43.856089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:37903 (local address 127.9.93.1:38335)
0928 16:49:43.856349 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:43.856358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:43.857078 (+ 720us) server_negotiation.cc:407] Connection header received
0928 16:49:43.857421 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:43.857427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:43.857540 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:43.857613 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:49:43.858565 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.859703 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.862991 (+ 3288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.864352 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.864474 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:43.864586 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:43.865225 (+ 639us) server_negotiation.cc:299] Negotiation successful
0928 16:49:43.865390 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 16:49:43.865993 11841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:43.857496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:59021 (local address 127.9.93.3:41059)
0928 16:49:43.857714 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:43.857721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:43.858412 (+ 691us) server_negotiation.cc:407] Connection header received
0928 16:49:43.858637 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:43.858642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:43.858723 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:43.858830 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:49:43.859761 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.860893 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.863438 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.864790 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.864886 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:43.865012 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:43.865581 (+ 569us) server_negotiation.cc:299] Negotiation successful
0928 16:49:43.865736 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 16:49:43.866540 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:43.856021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:38335 (local address 127.9.93.2:37903)
0928 16:49:43.856942 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:43.856964 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:43.857226 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:43.857868 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:43.857873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:43.857904 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:49:43.858383 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:43.858400 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.860357 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.860363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:43.862750 (+ 2387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:43.862776 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.864799 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.864805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:43.865008 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:43.865048 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 16:49:43.865997 (+ 949us) client_negotiation.cc:240] Negotiation successful
0928 16:49:43.866171 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":515,"threads_started":1}
I20260928 16:49:43.866801 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:43.857039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:41059 (local address 127.9.93.2:59021)
0928 16:49:43.858263 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:43.858280 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:43.858432 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:43.859035 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:43.859039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:43.859066 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:49:43.859541 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:43.859556 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.861125 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.861130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:43.863220 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:43.863235 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.865208 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.865214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:43.865381 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:43.865410 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 16:49:43.866244 (+ 834us) client_negotiation.cc:240] Negotiation successful
0928 16:49:43.866396 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1102,"thread_start_us":831,"threads_started":1}
I20260928 16:49:43.867899 11775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53ded63cf5c4e29b1464019f190e126" candidate_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e2fe45943745edb3479645dd18575c" is_pre_election: true
I20260928 16:49:43.867924 11495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53ded63cf5c4e29b1464019f190e126" candidate_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9567e6c8c4048d38639fbc991c3e211" is_pre_election: true
I20260928 16:49:43.868494 11775 raft_consensus.cc:2454] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed4b3cc6d19f4a6d9d4b126b8ed6d182 in term 0.
I20260928 16:49:43.868643 11495 raft_consensus.cc:2454] T f53ded63cf5c4e29b1464019f190e126 P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed4b3cc6d19f4a6d9d4b126b8ed6d182 in term 0.
I20260928 16:49:43.869410 11568 leader_election.cc:302] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ed4b3cc6d19f4a6d9d4b126b8ed6d182, f8e2fe45943745edb3479645dd18575c; no voters:
I20260928 16:49:43.870136 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:43.859253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:49831 (local address 127.9.93.1:38335)
0928 16:49:43.860376 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:43.860382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:43.860915 (+ 533us) server_negotiation.cc:407] Connection header received
0928 16:49:43.861193 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:43.861198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:43.861272 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:43.861345 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:49:43.862583 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.864135 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.866989 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.868427 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.868532 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:43.868701 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:43.869652 (+ 951us) server_negotiation.cc:299] Negotiation successful
0928 16:49:43.869804 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":978,"thread_start_us":666,"threads_started":1}
I20260928 16:49:43.870149 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:43.859176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:38335 (local address 127.9.93.3:49831)
0928 16:49:43.860820 (+ 1644us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:43.860843 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:43.861017 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:43.861894 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:43.861899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:43.861933 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:49:43.862342 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:43.862355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.864358 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.864364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:43.866750 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:43.866774 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.869304 (+ 2530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.869310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:43.869497 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:43.869530 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 16:49:43.869622 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 16:49:43.869782 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1498,"thread_start_us":1399,"threads_started":1}
I20260928 16:49:43.870386 11851 raft_consensus.cc:2781] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:49:43.870577 11851 raft_consensus.cc:482] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:49:43.870707 11851 raft_consensus.cc:3037] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:43.871142 11495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53ded63cf5c4e29b1464019f190e126" candidate_uuid: "f8e2fe45943745edb3479645dd18575c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9567e6c8c4048d38639fbc991c3e211" is_pre_election: true
I20260928 16:49:43.871399 11495 raft_consensus.cc:2454] T f53ded63cf5c4e29b1464019f190e126 P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e2fe45943745edb3479645dd18575c in term 0.
I20260928 16:49:43.871867 11708 leader_election.cc:302] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9567e6c8c4048d38639fbc991c3e211, f8e2fe45943745edb3479645dd18575c; no voters:
I20260928 16:49:43.872304 11852 raft_consensus.cc:2781] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:49:43.872462 11852 raft_consensus.cc:482] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:49:43.872608 11852 raft_consensus.cc:3037] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:43.873960 11851 raft_consensus.cc:504] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } }
I20260928 16:49:43.874825 11851 leader_election.cc:288] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [CANDIDATE]: Term 1 election: Requested vote from peers d9567e6c8c4048d38639fbc991c3e211 (127.9.93.1:38335), f8e2fe45943745edb3479645dd18575c (127.9.93.3:41059)
I20260928 16:49:43.875437 11775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53ded63cf5c4e29b1464019f190e126" candidate_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e2fe45943745edb3479645dd18575c"
I20260928 16:49:43.875592 11857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:43.861023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:41673 (local address 127.9.93.3:59609)
0928 16:49:43.868075 (+ 7052us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:43.868096 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:43.868310 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:43.868840 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:43.868845 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:43.868873 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:49:43.869244 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:43.869257 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.870664 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.870669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:43.872772 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:43.872793 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.874580 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.874588 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:43.874813 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:43.874856 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 16:49:43.875030 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 16:49:43.875191 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6829,"thread_start_us":8333,"threads_started":1}
I20260928 16:49:43.875773 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:43.861413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:59609 (local address 127.9.93.2:41673)
0928 16:49:43.861755 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:43.861762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:43.868192 (+ 6430us) server_negotiation.cc:407] Connection header received
0928 16:49:43.868474 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:43.868479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:43.868566 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:43.868635 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:49:43.869435 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.870466 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.872979 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.874270 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.874436 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:43.874598 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:43.875347 (+ 749us) server_negotiation.cc:299] Negotiation successful
0928 16:49:43.875501 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260928 16:49:43.876210 11495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53ded63cf5c4e29b1464019f190e126" candidate_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9567e6c8c4048d38639fbc991c3e211"
I20260928 16:49:43.876477 11495 raft_consensus.cc:3037] T f53ded63cf5c4e29b1464019f190e126 P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:43.876598 11852 raft_consensus.cc:504] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } }
I20260928 16:49:43.876840 11635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53ded63cf5c4e29b1464019f190e126" candidate_uuid: "f8e2fe45943745edb3479645dd18575c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" is_pre_election: true
I20260928 16:49:43.877142 11775 raft_consensus.cc:2379] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ed4b3cc6d19f4a6d9d4b126b8ed6d182 in current term 1: Already voted for candidate f8e2fe45943745edb3479645dd18575c in this term.
I20260928 16:49:43.877425 11635 raft_consensus.cc:2379] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e2fe45943745edb3479645dd18575c in current term 1: Already voted for candidate ed4b3cc6d19f4a6d9d4b126b8ed6d182 in this term.
I20260928 16:49:43.877417 11852 leader_election.cc:288] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c [CANDIDATE]: Term 1 election: Requested vote from peers d9567e6c8c4048d38639fbc991c3e211 (127.9.93.1:38335), ed4b3cc6d19f4a6d9d4b126b8ed6d182 (127.9.93.2:41673)
I20260928 16:49:43.878697 11494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53ded63cf5c4e29b1464019f190e126" candidate_uuid: "f8e2fe45943745edb3479645dd18575c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9567e6c8c4048d38639fbc991c3e211"
I20260928 16:49:43.878790 11635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53ded63cf5c4e29b1464019f190e126" candidate_uuid: "f8e2fe45943745edb3479645dd18575c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182"
I20260928 16:49:43.879052 11635 raft_consensus.cc:2379] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e2fe45943745edb3479645dd18575c in current term 1: Already voted for candidate ed4b3cc6d19f4a6d9d4b126b8ed6d182 in this term.
I20260928 16:49:43.879658 11708 leader_election.cc:302] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e2fe45943745edb3479645dd18575c; no voters: d9567e6c8c4048d38639fbc991c3e211, ed4b3cc6d19f4a6d9d4b126b8ed6d182
I20260928 16:49:43.879935 11495 raft_consensus.cc:2454] T f53ded63cf5c4e29b1464019f190e126 P d9567e6c8c4048d38639fbc991c3e211 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed4b3cc6d19f4a6d9d4b126b8ed6d182 in term 1.
I20260928 16:49:43.879964 11852 raft_consensus.cc:2726] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 16:49:43.880299 11568 leader_election.cc:302] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d9567e6c8c4048d38639fbc991c3e211, ed4b3cc6d19f4a6d9d4b126b8ed6d182; no voters: f8e2fe45943745edb3479645dd18575c
I20260928 16:49:43.880658 11851 raft_consensus.cc:2781] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:43.881120 11851 raft_consensus.cc:686] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 1 LEADER]: Becoming Leader. State: Replica: ed4b3cc6d19f4a6d9d4b126b8ed6d182, State: Running, Role: LEADER
I20260928 16:49:43.881682 11851 consensus_queue.cc:237] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } }
I20260928 16:49:43.887296 11337 catalog_manager.cc:5495] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed4b3cc6d19f4a6d9d4b126b8ed6d182 (127.9.93.2). New cstate: current_term: 1 leader_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } health_report { overall_health: HEALTHY } } }
Sep 28 16:49:43 dist-test-slave-wzcb krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614181, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:49:43.972939 11865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:43.958006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:45277 (local address 127.0.0.1:53042)
0928 16:49:43.959034 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:43.959058 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:43.959245 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:43.959571 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:43.959575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:43.960443 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:43.960747 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:43.960757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.961916 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.961920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:43.963008 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:43.963021 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.963867 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.963872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:43.964003 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:43.964669 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:43.964686 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:43.967062 (+ 2376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:43.970227 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:43.970244 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:43.970251 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:43.970752 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:43.971141 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:43.971144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:43.971147 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:43.971325 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:43.971944 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:43.971952 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:43.972139 (+ 187us) client_negotiation.cc:769] Sending connection context
0928 16:49:43.972293 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 16:49:43.972564 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"thread_start_us":395,"threads_started":1}
I20260928 16:49:43.973078 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:43.958071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53042 (local address 127.9.93.62:45277)
0928 16:49:43.959039 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:43.959044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:43.959135 (+ 91us) server_negotiation.cc:407] Connection header received
0928 16:49:43.959366 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:43.959370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:43.959437 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:43.959530 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:43.960923 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.961747 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.963163 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:43.963675 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:43.963769 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:43.967373 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:43.967382 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:43.967388 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:43.967426 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:43.970064 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:43.970865 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:43.970869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:43.970871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:43.971025 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:43.971431 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:43.971435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:43.971437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:43.971770 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:43.971884 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:43.972397 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 16:49:43.972726 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":855,"thread_start_us":524,"threads_started":1}
I20260928 16:49:43.978463 11337 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:53042:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 16:49:43.979598 11337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:49:43.987391 11615 tablet_service.cc:1467] Processing CreateTablet for tablet 076c08e937f545aca49649c4b8cde05d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d89ed9175c5847ed9a9d371b0b699090]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:49:43.987442 11755 tablet_service.cc:1467] Processing CreateTablet for tablet 076c08e937f545aca49649c4b8cde05d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d89ed9175c5847ed9a9d371b0b699090]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:49:43.987272 11475 tablet_service.cc:1467] Processing CreateTablet for tablet 076c08e937f545aca49649c4b8cde05d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d89ed9175c5847ed9a9d371b0b699090]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:49:43.988459 11615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076c08e937f545aca49649c4b8cde05d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:43.988462 11755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076c08e937f545aca49649c4b8cde05d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:43.988461 11475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076c08e937f545aca49649c4b8cde05d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:43.996241 11847 tablet_bootstrap.cc:492] T 076c08e937f545aca49649c4b8cde05d P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Bootstrap starting.
I20260928 16:49:43.996543 11846 tablet_bootstrap.cc:492] T 076c08e937f545aca49649c4b8cde05d P f8e2fe45943745edb3479645dd18575c: Bootstrap starting.
I20260928 16:49:43.997290 11844 tablet_bootstrap.cc:492] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211: Bootstrap starting.
I20260928 16:49:43.998412 11847 tablet_bootstrap.cc:654] T 076c08e937f545aca49649c4b8cde05d P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:43.998454 11846 tablet_bootstrap.cc:654] T 076c08e937f545aca49649c4b8cde05d P f8e2fe45943745edb3479645dd18575c: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:43.999106 11844 tablet_bootstrap.cc:654] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:44.000861 11846 tablet_bootstrap.cc:492] T 076c08e937f545aca49649c4b8cde05d P f8e2fe45943745edb3479645dd18575c: No bootstrap required, opened a new log
I20260928 16:49:44.001041 11846 ts_tablet_manager.cc:1397] T 076c08e937f545aca49649c4b8cde05d P f8e2fe45943745edb3479645dd18575c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 16:49:44.001461 11844 tablet_bootstrap.cc:492] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211: No bootstrap required, opened a new log
I20260928 16:49:44.001638 11844 ts_tablet_manager.cc:1397] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260928 16:49:44.001945 11846 raft_consensus.cc:348] T 076c08e937f545aca49649c4b8cde05d P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:44.002192 11846 raft_consensus.cc:374] T 076c08e937f545aca49649c4b8cde05d P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:44.002255 11847 tablet_bootstrap.cc:492] T 076c08e937f545aca49649c4b8cde05d P ed4b3cc6d19f4a6d9d4b126b8ed6d182: No bootstrap required, opened a new log
I20260928 16:49:44.002288 11846 raft_consensus.cc:729] T 076c08e937f545aca49649c4b8cde05d P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e2fe45943745edb3479645dd18575c, State: Initialized, Role: FOLLOWER
I20260928 16:49:44.002424 11847 ts_tablet_manager.cc:1397] T 076c08e937f545aca49649c4b8cde05d P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:49:44.002507 11844 raft_consensus.cc:348] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:44.002549 11846 consensus_queue.cc:260] T 076c08e937f545aca49649c4b8cde05d P f8e2fe45943745edb3479645dd18575c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:44.002769 11844 raft_consensus.cc:374] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:44.002857 11844 raft_consensus.cc:729] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9567e6c8c4048d38639fbc991c3e211, State: Initialized, Role: FOLLOWER
I20260928 16:49:44.003185 11846 ts_tablet_manager.cc:1428] T 076c08e937f545aca49649c4b8cde05d P f8e2fe45943745edb3479645dd18575c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 16:49:44.003185 11844 consensus_queue.cc:260] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:44.003419 11847 raft_consensus.cc:348] T 076c08e937f545aca49649c4b8cde05d P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:44.003683 11847 raft_consensus.cc:374] T 076c08e937f545aca49649c4b8cde05d P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:44.003783 11847 raft_consensus.cc:729] T 076c08e937f545aca49649c4b8cde05d P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed4b3cc6d19f4a6d9d4b126b8ed6d182, State: Initialized, Role: FOLLOWER
I20260928 16:49:44.003721 11850 tablet_replica.cc:439] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } } }
I20260928 16:49:44.004117 11850 tablet_replica.cc:442] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:49:44.004047 11847 consensus_queue.cc:260] T 076c08e937f545aca49649c4b8cde05d P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:44.004686 11847 ts_tablet_manager.cc:1428] T 076c08e937f545aca49649c4b8cde05d P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 16:49:44.005347 11844 ts_tablet_manager.cc:1428] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 16:49:44.005147 11851 tablet_replica.cc:439] T 076c08e937f545aca49649c4b8cde05d P ed4b3cc6d19f4a6d9d4b126b8ed6d182: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } } }
I20260928 16:49:44.005492 11851 tablet_replica.cc:442] T 076c08e937f545aca49649c4b8cde05d P ed4b3cc6d19f4a6d9d4b126b8ed6d182: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:49:44.005729 11852 tablet_replica.cc:439] T 076c08e937f545aca49649c4b8cde05d P f8e2fe45943745edb3479645dd18575c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } } }
I20260928 16:49:44.006139 11852 tablet_replica.cc:442] T 076c08e937f545aca49649c4b8cde05d P f8e2fe45943745edb3479645dd18575c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:49:44.291428 11850 raft_consensus.cc:482] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:49:44.291710 11850 raft_consensus.cc:504] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:44.293382 11850 leader_election.cc:288] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed4b3cc6d19f4a6d9d4b126b8ed6d182 (127.9.93.2:41673), f8e2fe45943745edb3479645dd18575c (127.9.93.3:41059)
I20260928 16:49:44.302201 11841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:44.294651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:40727 (local address 127.9.93.3:41059)
0928 16:49:44.294851 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:44.294858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:44.295355 (+ 497us) server_negotiation.cc:407] Connection header received
0928 16:49:44.295594 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:44.295601 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:44.295711 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:44.295823 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:49:44.296329 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:44.297280 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:44.299863 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:44.301105 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:44.301227 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:44.301320 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:44.301710 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 16:49:44.301865 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:49:44.302201 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:44.293821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:34815 (local address 127.9.93.2:41673)
0928 16:49:44.294063 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:44.294071 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:44.294377 (+ 306us) server_negotiation.cc:407] Connection header received
0928 16:49:44.294611 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:44.294619 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:44.294726 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:44.294837 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:49:44.295699 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:44.296773 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:44.299693 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:44.300963 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:44.301122 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:44.301259 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:44.301717 (+ 458us) server_negotiation.cc:299] Negotiation successful
0928 16:49:44.301868 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 16:49:44.302402 11870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:44.293714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:41673 (local address 127.9.93.1:34815)
0928 16:49:44.294254 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:44.294272 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:44.294432 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:44.294970 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:44.294975 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:44.295013 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:49:44.295448 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:44.295460 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:44.296939 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:44.296944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:44.299496 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:44.299517 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:44.301238 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:44.301246 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:44.301518 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:44.301570 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 16:49:44.301674 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 16:49:44.301868 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":73,"thread_start_us":272,"threads_started":1}
I20260928 16:49:44.302423 11871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:44.294488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:41059 (local address 127.9.93.1:40727)
0928 16:49:44.295222 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:44.295241 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:44.295451 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:44.295856 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:44.295861 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:44.295888 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:49:44.296177 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:44.296185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:44.297404 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:44.297414 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:44.299712 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:44.299723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:44.301321 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:44.301326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:44.301500 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:44.301576 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 16:49:44.301674 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 16:49:44.301868 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":77,"thread_start_us":385,"threads_started":1}
I20260928 16:49:44.303329 11775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076c08e937f545aca49649c4b8cde05d" candidate_uuid: "d9567e6c8c4048d38639fbc991c3e211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e2fe45943745edb3479645dd18575c" is_pre_election: true
I20260928 16:49:44.303615 11775 raft_consensus.cc:2454] T 076c08e937f545aca49649c4b8cde05d P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9567e6c8c4048d38639fbc991c3e211 in term 0.
I20260928 16:49:44.303854 11635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076c08e937f545aca49649c4b8cde05d" candidate_uuid: "d9567e6c8c4048d38639fbc991c3e211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" is_pre_election: true
I20260928 16:49:44.304183 11635 raft_consensus.cc:2454] T 076c08e937f545aca49649c4b8cde05d P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9567e6c8c4048d38639fbc991c3e211 in term 0.
I20260928 16:49:44.304140 11427 leader_election.cc:302] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9567e6c8c4048d38639fbc991c3e211, f8e2fe45943745edb3479645dd18575c; no voters:
I20260928 16:49:44.304584 11850 raft_consensus.cc:2781] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:49:44.304713 11850 raft_consensus.cc:482] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:49:44.304821 11850 raft_consensus.cc:3037] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:44.307420 11850 raft_consensus.cc:504] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:44.308100 11850 leader_election.cc:288] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211 [CANDIDATE]: Term 1 election: Requested vote from peers ed4b3cc6d19f4a6d9d4b126b8ed6d182 (127.9.93.2:41673), f8e2fe45943745edb3479645dd18575c (127.9.93.3:41059)
I20260928 16:49:44.308745 11635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076c08e937f545aca49649c4b8cde05d" candidate_uuid: "d9567e6c8c4048d38639fbc991c3e211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182"
I20260928 16:49:44.308851 11775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076c08e937f545aca49649c4b8cde05d" candidate_uuid: "d9567e6c8c4048d38639fbc991c3e211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e2fe45943745edb3479645dd18575c"
I20260928 16:49:44.308979 11635 raft_consensus.cc:3037] T 076c08e937f545aca49649c4b8cde05d P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:44.309108 11775 raft_consensus.cc:3037] T 076c08e937f545aca49649c4b8cde05d P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:44.311573 11635 raft_consensus.cc:2454] T 076c08e937f545aca49649c4b8cde05d P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9567e6c8c4048d38639fbc991c3e211 in term 1.
I20260928 16:49:44.311614 11775 raft_consensus.cc:2454] T 076c08e937f545aca49649c4b8cde05d P f8e2fe45943745edb3479645dd18575c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9567e6c8c4048d38639fbc991c3e211 in term 1.
I20260928 16:49:44.312182 11427 leader_election.cc:302] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9567e6c8c4048d38639fbc991c3e211, f8e2fe45943745edb3479645dd18575c; no voters:
I20260928 16:49:44.312573 11850 raft_consensus.cc:2781] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:44.313040 11850 raft_consensus.cc:686] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211 [term 1 LEADER]: Becoming Leader. State: Replica: d9567e6c8c4048d38639fbc991c3e211, State: Running, Role: LEADER
I20260928 16:49:44.313585 11850 consensus_queue.cc:237] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:44.317423 11872 tablet_replica.cc:439] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211: TxnStatusTablet state changed. Reason: New leader d9567e6c8c4048d38639fbc991c3e211. Latest consensus state: current_term: 1 leader_uuid: "d9567e6c8c4048d38639fbc991c3e211" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } } }
I20260928 16:49:44.317932 11872 tablet_replica.cc:442] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:49:44.318827 11874 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 16:49:44.319162 11874 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 16:49:44.319118 11335 catalog_manager.cc:5495] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9567e6c8c4048d38639fbc991c3e211 (127.9.93.1). New cstate: current_term: 1 leader_uuid: "d9567e6c8c4048d38639fbc991c3e211" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } health_report { overall_health: UNKNOWN } } }
I20260928 16:49:44.376024 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:44.369529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48530 (local address 127.9.93.1:38335)
0928 16:49:44.369761 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:44.369768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:44.370007 (+ 239us) server_negotiation.cc:407] Connection header received
0928 16:49:44.370349 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:44.370353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:44.370472 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:44.370650 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:44.371119 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:44.372015 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:44.373488 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:44.373986 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:44.374109 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:44.374671 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:44.375283 (+ 612us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:44.375477 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:44.375576 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 16:49:44.375693 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 16:49:44.376065 11865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:44.369521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:38335 (local address 127.0.0.1:48530)
0928 16:49:44.369906 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:44.369922 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:44.370116 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:44.370775 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:44.370780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:44.370796 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:49:44.370975 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:44.370984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:44.372178 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:44.372184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:44.373320 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:44.373337 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:44.374163 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:44.374170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:44.374380 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:44.374453 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:49:44.375435 (+ 982us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:49:44.375467 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 16:49:44.375555 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 16:49:44.375745 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229}
I20260928 16:49:44.385655 11635 raft_consensus.cc:1261] T 076c08e937f545aca49649c4b8cde05d P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 1 FOLLOWER]: Refusing update from remote peer d9567e6c8c4048d38639fbc991c3e211: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:49:44.385710 11775 raft_consensus.cc:1261] T 076c08e937f545aca49649c4b8cde05d P f8e2fe45943745edb3479645dd18575c [term 1 FOLLOWER]: Refusing update from remote peer d9567e6c8c4048d38639fbc991c3e211: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:49:44.386832 11872 consensus_queue.cc:1035] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:49:44.387353 11850 consensus_queue.cc:1035] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:49:44.400221 11851 tablet_replica.cc:439] T 076c08e937f545aca49649c4b8cde05d P ed4b3cc6d19f4a6d9d4b126b8ed6d182: TxnStatusTablet state changed. Reason: New leader d9567e6c8c4048d38639fbc991c3e211. Latest consensus state: current_term: 1 leader_uuid: "d9567e6c8c4048d38639fbc991c3e211" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } } }
I20260928 16:49:44.400573 11851 tablet_replica.cc:442] T 076c08e937f545aca49649c4b8cde05d P ed4b3cc6d19f4a6d9d4b126b8ed6d182: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:49:44.402921 11877 mvcc.cc:205] Tried to move back new op lower bound from 7334355699231195136 to 7334355698952978432. Current Snapshot: MvccSnapshot[applied={T|T < 7334355699231195136}]
I20260928 16:49:44.404855 11852 tablet_replica.cc:439] T 076c08e937f545aca49649c4b8cde05d P f8e2fe45943745edb3479645dd18575c: TxnStatusTablet state changed. Reason: New leader d9567e6c8c4048d38639fbc991c3e211. Latest consensus state: current_term: 1 leader_uuid: "d9567e6c8c4048d38639fbc991c3e211" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } } }
I20260928 16:49:44.405193 11852 tablet_replica.cc:442] T 076c08e937f545aca49649c4b8cde05d P f8e2fe45943745edb3479645dd18575c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:49:44.412141 11872 tablet_replica.cc:439] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9567e6c8c4048d38639fbc991c3e211" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } } }
I20260928 16:49:44.412513 11872 tablet_replica.cc:442] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:49:44.412949 11872 tablet_replica.cc:439] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9567e6c8c4048d38639fbc991c3e211" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } } }
I20260928 16:49:44.413257 11872 tablet_replica.cc:442] T 076c08e937f545aca49649c4b8cde05d P d9567e6c8c4048d38639fbc991c3e211: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:49:44.420056 11851 tablet_replica.cc:439] T 076c08e937f545aca49649c4b8cde05d P ed4b3cc6d19f4a6d9d4b126b8ed6d182: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9567e6c8c4048d38639fbc991c3e211" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } } }
I20260928 16:49:44.420444 11851 tablet_replica.cc:442] T 076c08e937f545aca49649c4b8cde05d P ed4b3cc6d19f4a6d9d4b126b8ed6d182: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:49:44.425913 11851 consensus_queue.cc:1035] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:49:44.444648 11852 tablet_replica.cc:439] T 076c08e937f545aca49649c4b8cde05d P f8e2fe45943745edb3479645dd18575c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9567e6c8c4048d38639fbc991c3e211" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } } }
I20260928 16:49:44.445103 11852 tablet_replica.cc:442] T 076c08e937f545aca49649c4b8cde05d P f8e2fe45943745edb3479645dd18575c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:49:44.446483 11884 consensus_queue.cc:1035] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:49:44.454952 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:44.434572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60970 (local address 127.9.93.2:41673)
0928 16:49:44.434788 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:44.434795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:44.443960 (+ 9165us) server_negotiation.cc:407] Connection header received
0928 16:49:44.446659 (+ 2699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:44.446666 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:44.446808 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:44.446959 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:44.447923 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:44.449033 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:44.451542 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:44.452247 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:44.452373 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:44.453162 (+ 789us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:44.454067 (+ 905us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:44.454247 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:44.454525 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 16:49:44.454646 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 16:49:44.484620 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:44.478212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:47765 (local address 127.9.93.1:38335)
0928 16:49:44.478458 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:44.478466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:44.479166 (+ 700us) server_negotiation.cc:407] Connection header received
0928 16:49:44.479425 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:44.479431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:44.479508 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:44.479591 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:44.480133 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:44.481002 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:44.482504 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:44.483009 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:44.483132 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:44.483737 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:44.483907 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:44.483998 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:44.484223 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 16:49:44.484331 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260928 16:49:44.485409 11892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:44.478212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:38335 (local address 127.9.93.2:47765)
0928 16:49:44.479042 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:44.479060 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:44.479248 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:44.479628 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:44.479633 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:44.479645 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:49:44.479967 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:44.479978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:44.481179 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:44.481185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:44.482339 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:44.482354 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:44.483223 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:44.483230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:44.483429 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:44.483507 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:49:44.484061 (+ 554us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:49:44.484074 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 16:49:44.484929 (+ 855us) client_negotiation.cc:240] Negotiation successful
0928 16:49:44.485089 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":450,"threads_started":1}
I20260928 16:49:44.547333 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:44.537181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:46681 (local address 127.9.93.2:41673)
0928 16:49:44.537407 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:44.537415 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:44.540079 (+ 2664us) server_negotiation.cc:407] Connection header received
0928 16:49:44.540388 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:44.540395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:44.540584 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:44.540697 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:44.541718 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:44.542769 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:44.545123 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:44.545685 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:44.545828 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:44.546398 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:44.546553 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:44.546650 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:44.546837 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 16:49:44.546943 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 16:49:44.547487 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:44.537129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:41673 (local address 127.9.93.1:46681)
0928 16:49:44.539903 (+ 2774us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:44.539930 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:44.540149 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:44.541003 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:44.541009 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:44.541026 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:49:44.541451 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:44.541467 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:44.542926 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:44.542932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:44.544917 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:44.544941 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:44.545913 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:44.545921 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:44.546093 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:44.546168 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:49:44.546685 (+ 517us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:49:44.546700 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 16:49:44.546796 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 16:49:44.546989 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2548,"thread_start_us":2379,"threads_started":1}
I20260928 16:49:44.603345 11335 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:49:44.603752 11335 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:49:44.608599 11335 catalog_manager.cc:5782] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6: Sending DeleteTablet for 3 replicas of tablet f53ded63cf5c4e29b1464019f190e126
I20260928 16:49:44.610251 11615 tablet_service.cc:1514] Processing DeleteTablet for tablet f53ded63cf5c4e29b1464019f190e126 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:49:44 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:60960
I20260928 16:49:44.611654 11755 tablet_service.cc:1514] Processing DeleteTablet for tablet f53ded63cf5c4e29b1464019f190e126 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:49:44 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:48150
I20260928 16:49:44.611536 11336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:49:44.612506 11336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:49:44.614594 11901 tablet_replica.cc:330] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c: stopping tablet replica
I20260928 16:49:44.615331 11901 raft_consensus.cc:2229] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:49:44.615898 11901 raft_consensus.cc:2258] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:49:44.618494 11901 ts_tablet_manager.cc:1905] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:49:44.626464 11614 tablet_service.cc:1467] Processing CreateTablet for tablet 5e12646da5454f12aa38132ad4362d2e (DEFAULT_TABLE table=test-table [id=fc89584ce25a40bdb8ee3e591ada4410]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:49:44.627244 11614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e12646da5454f12aa38132ad4362d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:44.627848 11901 ts_tablet_manager.cc:1918] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:49:44.628060 11901 log.cc:1196] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/f53ded63cf5c4e29b1464019f190e126
I20260928 16:49:44.628033 11755 tablet_service.cc:1467] Processing CreateTablet for tablet 5e12646da5454f12aa38132ad4362d2e (DEFAULT_TABLE table=test-table [id=fc89584ce25a40bdb8ee3e591ada4410]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:49:44.628561 11755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e12646da5454f12aa38132ad4362d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:44.628815 11474 tablet_service.cc:1467] Processing CreateTablet for tablet 5e12646da5454f12aa38132ad4362d2e (DEFAULT_TABLE table=test-table [id=fc89584ce25a40bdb8ee3e591ada4410]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:49:44.629369 11474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e12646da5454f12aa38132ad4362d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:44.632783 11475 tablet_service.cc:1514] Processing DeleteTablet for tablet f53ded63cf5c4e29b1464019f190e126 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:49:44 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:48528
I20260928 16:49:44.636543 11901 ts_tablet_manager.cc:1939] T f53ded63cf5c4e29b1464019f190e126 P f8e2fe45943745edb3479645dd18575c: Deleting consensus metadata
I20260928 16:49:44.638801 11324 catalog_manager.cc:4841] TS f8e2fe45943745edb3479645dd18575c (127.9.93.3:41059): tablet f53ded63cf5c4e29b1464019f190e126 (table test-table [id=09fc40cce8394521a5002861136883cb]) successfully deleted
I20260928 16:49:44.644191 11905 tablet_bootstrap.cc:492] T 5e12646da5454f12aa38132ad4362d2e P d9567e6c8c4048d38639fbc991c3e211: Bootstrap starting.
I20260928 16:49:44.645557 11900 tablet_replica.cc:330] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182: stopping tablet replica
I20260928 16:49:44.645699 11904 tablet_bootstrap.cc:492] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Bootstrap starting.
I20260928 16:49:44.646347 11900 raft_consensus.cc:2229] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:49:44.647432 11900 raft_consensus.cc:2258] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:49:44.648794 11904 tablet_bootstrap.cc:654] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:44.645087 11903 tablet_replica.cc:330] T f53ded63cf5c4e29b1464019f190e126 P d9567e6c8c4048d38639fbc991c3e211: stopping tablet replica
I20260928 16:49:44.649916 11902 tablet_bootstrap.cc:492] T 5e12646da5454f12aa38132ad4362d2e P f8e2fe45943745edb3479645dd18575c: Bootstrap starting.
I20260928 16:49:44.651556 11905 tablet_bootstrap.cc:654] T 5e12646da5454f12aa38132ad4362d2e P d9567e6c8c4048d38639fbc991c3e211: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:44.652065 11900 ts_tablet_manager.cc:1905] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:49:44.652377 11902 tablet_bootstrap.cc:654] T 5e12646da5454f12aa38132ad4362d2e P f8e2fe45943745edb3479645dd18575c: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:44.657430 11903 raft_consensus.cc:2229] T f53ded63cf5c4e29b1464019f190e126 P d9567e6c8c4048d38639fbc991c3e211 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:49:44.658830 11903 raft_consensus.cc:2258] T f53ded63cf5c4e29b1464019f190e126 P d9567e6c8c4048d38639fbc991c3e211 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:49:44.662302 11903 ts_tablet_manager.cc:1905] T f53ded63cf5c4e29b1464019f190e126 P d9567e6c8c4048d38639fbc991c3e211: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:49:44.664745 11900 ts_tablet_manager.cc:1918] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:49:44.664853 11902 tablet_bootstrap.cc:492] T 5e12646da5454f12aa38132ad4362d2e P f8e2fe45943745edb3479645dd18575c: No bootstrap required, opened a new log
I20260928 16:49:44.665045 11902 ts_tablet_manager.cc:1397] T 5e12646da5454f12aa38132ad4362d2e P f8e2fe45943745edb3479645dd18575c: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.003s
I20260928 16:49:44.665133 11900 log.cc:1196] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/f53ded63cf5c4e29b1464019f190e126
I20260928 16:49:44.663887 11905 tablet_bootstrap.cc:492] T 5e12646da5454f12aa38132ad4362d2e P d9567e6c8c4048d38639fbc991c3e211: No bootstrap required, opened a new log
I20260928 16:49:44.666066 11900 ts_tablet_manager.cc:1939] T f53ded63cf5c4e29b1464019f190e126 P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Deleting consensus metadata
I20260928 16:49:44.666208 11905 ts_tablet_manager.cc:1397] T 5e12646da5454f12aa38132ad4362d2e P d9567e6c8c4048d38639fbc991c3e211: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.008s
I20260928 16:49:44.667146 11902 raft_consensus.cc:348] T 5e12646da5454f12aa38132ad4362d2e P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:44.667269 11905 raft_consensus.cc:348] T 5e12646da5454f12aa38132ad4362d2e P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:44.667621 11905 raft_consensus.cc:374] T 5e12646da5454f12aa38132ad4362d2e P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:44.667654 11902 raft_consensus.cc:374] T 5e12646da5454f12aa38132ad4362d2e P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:44.667802 11905 raft_consensus.cc:729] T 5e12646da5454f12aa38132ad4362d2e P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9567e6c8c4048d38639fbc991c3e211, State: Initialized, Role: FOLLOWER
I20260928 16:49:44.667850 11902 raft_consensus.cc:729] T 5e12646da5454f12aa38132ad4362d2e P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e2fe45943745edb3479645dd18575c, State: Initialized, Role: FOLLOWER
I20260928 16:49:44.668229 11902 consensus_queue.cc:260] T 5e12646da5454f12aa38132ad4362d2e P f8e2fe45943745edb3479645dd18575c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:44.668241 11905 consensus_queue.cc:260] T 5e12646da5454f12aa38132ad4362d2e P d9567e6c8c4048d38639fbc991c3e211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:44.669243 11905 ts_tablet_manager.cc:1428] T 5e12646da5454f12aa38132ad4362d2e P d9567e6c8c4048d38639fbc991c3e211: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:49:44.670394 11324 catalog_manager.cc:4841] TS ed4b3cc6d19f4a6d9d4b126b8ed6d182 (127.9.93.2:41673): tablet f53ded63cf5c4e29b1464019f190e126 (table test-table [id=09fc40cce8394521a5002861136883cb]) successfully deleted
I20260928 16:49:44.671413 11904 tablet_bootstrap.cc:492] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182: No bootstrap required, opened a new log
I20260928 16:49:44.671619 11904 ts_tablet_manager.cc:1397] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20260928 16:49:44.671710 11902 ts_tablet_manager.cc:1428] T 5e12646da5454f12aa38132ad4362d2e P f8e2fe45943745edb3479645dd18575c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260928 16:49:44.680895 11903 ts_tablet_manager.cc:1918] T f53ded63cf5c4e29b1464019f190e126 P d9567e6c8c4048d38639fbc991c3e211: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:49:44.681190 11903 log.cc:1196] T f53ded63cf5c4e29b1464019f190e126 P d9567e6c8c4048d38639fbc991c3e211: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-0/wal/wals/f53ded63cf5c4e29b1464019f190e126
I20260928 16:49:44.682051 11903 ts_tablet_manager.cc:1939] T f53ded63cf5c4e29b1464019f190e126 P d9567e6c8c4048d38639fbc991c3e211: Deleting consensus metadata
I20260928 16:49:44.682219 11904 raft_consensus.cc:348] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:44.682646 11904 raft_consensus.cc:374] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:44.682828 11904 raft_consensus.cc:729] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed4b3cc6d19f4a6d9d4b126b8ed6d182, State: Initialized, Role: FOLLOWER
I20260928 16:49:44.683264 11904 consensus_queue.cc:260] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:44.684589 11324 catalog_manager.cc:4841] TS d9567e6c8c4048d38639fbc991c3e211 (127.9.93.1:38335): tablet f53ded63cf5c4e29b1464019f190e126 (table test-table [id=09fc40cce8394521a5002861136883cb]) successfully deleted
I20260928 16:49:44.685750 11904 ts_tablet_manager.cc:1428] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20260928 16:49:44.913371 11884 raft_consensus.cc:482] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:49:44.913746 11884 raft_consensus.cc:504] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:44.914819 11884 leader_election.cc:288] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9567e6c8c4048d38639fbc991c3e211 (127.9.93.1:38335), f8e2fe45943745edb3479645dd18575c (127.9.93.3:41059)
I20260928 16:49:44.915602 11495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e12646da5454f12aa38132ad4362d2e" candidate_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9567e6c8c4048d38639fbc991c3e211" is_pre_election: true
I20260928 16:49:44.915730 11774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e12646da5454f12aa38132ad4362d2e" candidate_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e2fe45943745edb3479645dd18575c" is_pre_election: true
I20260928 16:49:44.915927 11495 raft_consensus.cc:2454] T 5e12646da5454f12aa38132ad4362d2e P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed4b3cc6d19f4a6d9d4b126b8ed6d182 in term 0.
I20260928 16:49:44.916069 11774 raft_consensus.cc:2454] T 5e12646da5454f12aa38132ad4362d2e P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed4b3cc6d19f4a6d9d4b126b8ed6d182 in term 0.
I20260928 16:49:44.916570 11568 leader_election.cc:302] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9567e6c8c4048d38639fbc991c3e211, ed4b3cc6d19f4a6d9d4b126b8ed6d182; no voters:
I20260928 16:49:44.916934 11884 raft_consensus.cc:2781] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:49:44.917061 11884 raft_consensus.cc:482] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:49:44.917189 11884 raft_consensus.cc:3037] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:44.921028 11884 raft_consensus.cc:504] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:44.922048 11884 leader_election.cc:288] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [CANDIDATE]: Term 1 election: Requested vote from peers d9567e6c8c4048d38639fbc991c3e211 (127.9.93.1:38335), f8e2fe45943745edb3479645dd18575c (127.9.93.3:41059)
I20260928 16:49:44.922767 11495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e12646da5454f12aa38132ad4362d2e" candidate_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9567e6c8c4048d38639fbc991c3e211"
I20260928 16:49:44.922839 11774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e12646da5454f12aa38132ad4362d2e" candidate_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e2fe45943745edb3479645dd18575c"
I20260928 16:49:44.923090 11774 raft_consensus.cc:3037] T 5e12646da5454f12aa38132ad4362d2e P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:44.923123 11495 raft_consensus.cc:3037] T 5e12646da5454f12aa38132ad4362d2e P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:44.925592 11774 raft_consensus.cc:2454] T 5e12646da5454f12aa38132ad4362d2e P f8e2fe45943745edb3479645dd18575c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed4b3cc6d19f4a6d9d4b126b8ed6d182 in term 1.
I20260928 16:49:44.926347 11568 leader_election.cc:302] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ed4b3cc6d19f4a6d9d4b126b8ed6d182, f8e2fe45943745edb3479645dd18575c; no voters:
I20260928 16:49:44.926504 11495 raft_consensus.cc:2454] T 5e12646da5454f12aa38132ad4362d2e P d9567e6c8c4048d38639fbc991c3e211 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed4b3cc6d19f4a6d9d4b126b8ed6d182 in term 1.
I20260928 16:49:44.927125 11884 raft_consensus.cc:2781] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:44.927565 11884 raft_consensus.cc:686] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 1 LEADER]: Becoming Leader. State: Replica: ed4b3cc6d19f4a6d9d4b126b8ed6d182, State: Running, Role: LEADER
I20260928 16:49:44.927884 11884 consensus_queue.cc:237] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:44.931236 11336 catalog_manager.cc:5495] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed4b3cc6d19f4a6d9d4b126b8ed6d182 (127.9.93.2). New cstate: current_term: 1 leader_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } health_report { overall_health: UNKNOWN } } }
I20260928 16:49:45.026113 11495 raft_consensus.cc:1261] T 5e12646da5454f12aa38132ad4362d2e P d9567e6c8c4048d38639fbc991c3e211 [term 1 FOLLOWER]: Refusing update from remote peer ed4b3cc6d19f4a6d9d4b126b8ed6d182: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:49:45.026823 11774 raft_consensus.cc:1261] T 5e12646da5454f12aa38132ad4362d2e P f8e2fe45943745edb3479645dd18575c [term 1 FOLLOWER]: Refusing update from remote peer ed4b3cc6d19f4a6d9d4b126b8ed6d182: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:49:45.026821 11884 consensus_queue.cc:1035] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:49:45.027482 11884 consensus_queue.cc:1035] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:49:45.033560 11878 mvcc.cc:205] Tried to move back new op lower bound from 7334355701860261888 to 7334355701467484160. Current Snapshot: MvccSnapshot[applied={T|T < 7334355701860261888}]
W20260928 16:49:45.056739 11878 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 16:49:46.665342 11336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:49:46.665629 11336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:49:46.670814 11336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6: Sending DeleteTablet for 3 replicas of tablet 5e12646da5454f12aa38132ad4362d2e
I20260928 16:49:46.671896 11475 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e12646da5454f12aa38132ad4362d2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:49:46 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:48528
I20260928 16:49:46.671911 11614 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e12646da5454f12aa38132ad4362d2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:49:46 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:60960
I20260928 16:49:46.672013 11755 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e12646da5454f12aa38132ad4362d2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:49:46 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:48150
I20260928 16:49:46.684995 11937 tablet_replica.cc:330] T 5e12646da5454f12aa38132ad4362d2e P d9567e6c8c4048d38639fbc991c3e211: stopping tablet replica
I20260928 16:49:46.685479 11937 raft_consensus.cc:2229] T 5e12646da5454f12aa38132ad4362d2e P d9567e6c8c4048d38639fbc991c3e211 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:49:46.685894 11937 raft_consensus.cc:2258] T 5e12646da5454f12aa38132ad4362d2e P d9567e6c8c4048d38639fbc991c3e211 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:49:46.686664 11936 tablet_replica.cc:330] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182: stopping tablet replica
I20260928 16:49:46.686734 11938 tablet_replica.cc:330] T 5e12646da5454f12aa38132ad4362d2e P f8e2fe45943745edb3479645dd18575c: stopping tablet replica
I20260928 16:49:46.686914 11936 raft_consensus.cc:2229] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:49:46.687077 11938 raft_consensus.cc:2229] T 5e12646da5454f12aa38132ad4362d2e P f8e2fe45943745edb3479645dd18575c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:49:46.687548 11936 raft_consensus.cc:2258] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:49:46.687726 11937 ts_tablet_manager.cc:1905] T 5e12646da5454f12aa38132ad4362d2e P d9567e6c8c4048d38639fbc991c3e211: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:49:46.687798 11938 raft_consensus.cc:2258] T 5e12646da5454f12aa38132ad4362d2e P f8e2fe45943745edb3479645dd18575c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:49:46.688937 11936 ts_tablet_manager.cc:1905] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:49:46.689042 11938 ts_tablet_manager.cc:1905] T 5e12646da5454f12aa38132ad4362d2e P f8e2fe45943745edb3479645dd18575c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:49:46.696200 11937 ts_tablet_manager.cc:1918] T 5e12646da5454f12aa38132ad4362d2e P d9567e6c8c4048d38639fbc991c3e211: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 16:49:46.696470 11937 log.cc:1196] T 5e12646da5454f12aa38132ad4362d2e P d9567e6c8c4048d38639fbc991c3e211: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-0/wal/wals/5e12646da5454f12aa38132ad4362d2e
I20260928 16:49:46.696686 11936 ts_tablet_manager.cc:1918] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 16:49:46.696918 11936 log.cc:1196] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/5e12646da5454f12aa38132ad4362d2e
I20260928 16:49:46.697156 11937 ts_tablet_manager.cc:1939] T 5e12646da5454f12aa38132ad4362d2e P d9567e6c8c4048d38639fbc991c3e211: Deleting consensus metadata
I20260928 16:49:46.697182 11938 ts_tablet_manager.cc:1918] T 5e12646da5454f12aa38132ad4362d2e P f8e2fe45943745edb3479645dd18575c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 16:49:46.697456 11938 log.cc:1196] T 5e12646da5454f12aa38132ad4362d2e P f8e2fe45943745edb3479645dd18575c: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/5e12646da5454f12aa38132ad4362d2e
I20260928 16:49:46.697507 11936 ts_tablet_manager.cc:1939] T 5e12646da5454f12aa38132ad4362d2e P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Deleting consensus metadata
I20260928 16:49:46.698010 11938 ts_tablet_manager.cc:1939] T 5e12646da5454f12aa38132ad4362d2e P f8e2fe45943745edb3479645dd18575c: Deleting consensus metadata
I20260928 16:49:46.698832 11324 catalog_manager.cc:4841] TS d9567e6c8c4048d38639fbc991c3e211 (127.9.93.1:38335): tablet 5e12646da5454f12aa38132ad4362d2e (table test-table [id=fc89584ce25a40bdb8ee3e591ada4410]) successfully deleted
I20260928 16:49:46.699193 11324 catalog_manager.cc:4841] TS ed4b3cc6d19f4a6d9d4b126b8ed6d182 (127.9.93.2:41673): tablet 5e12646da5454f12aa38132ad4362d2e (table test-table [id=fc89584ce25a40bdb8ee3e591ada4410]) successfully deleted
I20260928 16:49:46.699445 11324 catalog_manager.cc:4841] TS f8e2fe45943745edb3479645dd18575c (127.9.93.3:41059): tablet 5e12646da5454f12aa38132ad4362d2e (table test-table [id=fc89584ce25a40bdb8ee3e591ada4410]) successfully deleted
Sep 28 16:49:46 dist-test-slave-wzcb krb5kdc[11292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614186, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 16:49:46 dist-test-slave-wzcb krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614186, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:49:46.738433 11946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:46.721481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53050 (local address 127.9.93.62:45277)
0928 16:49:46.722120 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:46.722127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:46.722216 (+ 89us) server_negotiation.cc:407] Connection header received
0928 16:49:46.722443 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:46.722447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:46.722518 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:46.722627 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:46.724290 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:46.725175 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:46.726614 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:46.727087 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:46.727256 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:46.731765 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:46.731771 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:46.731775 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:46.731822 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:46.735222 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:46.736204 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:46.736210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:46.736213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:46.736378 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:46.736820 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:46.736824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:46.736826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:46.737155 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:46.737429 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:46.737763 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 16:49:46.738038 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":302,"threads_started":1}
I20260928 16:49:46.742697 11336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:49:46.743449 11336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:49:46.751114 11475 tablet_service.cc:1467] Processing CreateTablet for tablet 14b277c80bac4ba3ba9764bf0873a04e (DEFAULT_TABLE table=test-table [id=4d29a8053413464ca17c4685bdf93969]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:49:46.751353 11614 tablet_service.cc:1467] Processing CreateTablet for tablet 14b277c80bac4ba3ba9764bf0873a04e (DEFAULT_TABLE table=test-table [id=4d29a8053413464ca17c4685bdf93969]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:49:46.751415 11755 tablet_service.cc:1467] Processing CreateTablet for tablet 14b277c80bac4ba3ba9764bf0873a04e (DEFAULT_TABLE table=test-table [id=4d29a8053413464ca17c4685bdf93969]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:49:46.751788 11475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b277c80bac4ba3ba9764bf0873a04e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:46.751868 11755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b277c80bac4ba3ba9764bf0873a04e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:46.751919 11614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b277c80bac4ba3ba9764bf0873a04e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:46.758733 11947 tablet_bootstrap.cc:492] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Bootstrap starting.
I20260928 16:49:46.759395 11948 tablet_bootstrap.cc:492] T 14b277c80bac4ba3ba9764bf0873a04e P d9567e6c8c4048d38639fbc991c3e211: Bootstrap starting.
I20260928 16:49:46.761129 11947 tablet_bootstrap.cc:654] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:46.761365 11948 tablet_bootstrap.cc:654] T 14b277c80bac4ba3ba9764bf0873a04e P d9567e6c8c4048d38639fbc991c3e211: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:46.762782 11949 tablet_bootstrap.cc:492] T 14b277c80bac4ba3ba9764bf0873a04e P f8e2fe45943745edb3479645dd18575c: Bootstrap starting.
I20260928 16:49:46.764535 11947 tablet_bootstrap.cc:492] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182: No bootstrap required, opened a new log
I20260928 16:49:46.764698 11947 ts_tablet_manager.cc:1397] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 16:49:46.764730 11949 tablet_bootstrap.cc:654] T 14b277c80bac4ba3ba9764bf0873a04e P f8e2fe45943745edb3479645dd18575c: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:46.765463 11948 tablet_bootstrap.cc:492] T 14b277c80bac4ba3ba9764bf0873a04e P d9567e6c8c4048d38639fbc991c3e211: No bootstrap required, opened a new log
I20260928 16:49:46.765617 11948 ts_tablet_manager.cc:1397] T 14b277c80bac4ba3ba9764bf0873a04e P d9567e6c8c4048d38639fbc991c3e211: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:49:46.765456 11947 raft_consensus.cc:348] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:46.765750 11947 raft_consensus.cc:374] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:46.765858 11947 raft_consensus.cc:729] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed4b3cc6d19f4a6d9d4b126b8ed6d182, State: Initialized, Role: FOLLOWER
I20260928 16:49:46.766124 11947 consensus_queue.cc:260] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:46.766516 11948 raft_consensus.cc:348] T 14b277c80bac4ba3ba9764bf0873a04e P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:46.766870 11948 raft_consensus.cc:374] T 14b277c80bac4ba3ba9764bf0873a04e P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:46.766995 11948 raft_consensus.cc:729] T 14b277c80bac4ba3ba9764bf0873a04e P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9567e6c8c4048d38639fbc991c3e211, State: Initialized, Role: FOLLOWER
I20260928 16:49:46.767272 11948 consensus_queue.cc:260] T 14b277c80bac4ba3ba9764bf0873a04e P d9567e6c8c4048d38639fbc991c3e211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:46.768137 11948 ts_tablet_manager.cc:1428] T 14b277c80bac4ba3ba9764bf0873a04e P d9567e6c8c4048d38639fbc991c3e211: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 16:49:46.768594 11949 tablet_bootstrap.cc:492] T 14b277c80bac4ba3ba9764bf0873a04e P f8e2fe45943745edb3479645dd18575c: No bootstrap required, opened a new log
I20260928 16:49:46.768715 11947 ts_tablet_manager.cc:1428] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 16:49:46.768872 11949 ts_tablet_manager.cc:1397] T 14b277c80bac4ba3ba9764bf0873a04e P f8e2fe45943745edb3479645dd18575c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 16:49:46.769769 11949 raft_consensus.cc:348] T 14b277c80bac4ba3ba9764bf0873a04e P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:46.770061 11949 raft_consensus.cc:374] T 14b277c80bac4ba3ba9764bf0873a04e P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:46.770174 11949 raft_consensus.cc:729] T 14b277c80bac4ba3ba9764bf0873a04e P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e2fe45943745edb3479645dd18575c, State: Initialized, Role: FOLLOWER
I20260928 16:49:46.770418 11949 consensus_queue.cc:260] T 14b277c80bac4ba3ba9764bf0873a04e P f8e2fe45943745edb3479645dd18575c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:46.772068 11949 ts_tablet_manager.cc:1428] T 14b277c80bac4ba3ba9764bf0873a04e P f8e2fe45943745edb3479645dd18575c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 16:49:47.146308 11931 raft_consensus.cc:482] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:49:47.146590 11931 raft_consensus.cc:504] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:47.147663 11931 leader_election.cc:288] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9567e6c8c4048d38639fbc991c3e211 (127.9.93.1:38335), f8e2fe45943745edb3479645dd18575c (127.9.93.3:41059)
I20260928 16:49:47.148362 11495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b277c80bac4ba3ba9764bf0873a04e" candidate_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9567e6c8c4048d38639fbc991c3e211" is_pre_election: true
I20260928 16:49:47.148542 11774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b277c80bac4ba3ba9764bf0873a04e" candidate_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e2fe45943745edb3479645dd18575c" is_pre_election: true
I20260928 16:49:47.148676 11495 raft_consensus.cc:2454] T 14b277c80bac4ba3ba9764bf0873a04e P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed4b3cc6d19f4a6d9d4b126b8ed6d182 in term 0.
I20260928 16:49:47.148866 11774 raft_consensus.cc:2454] T 14b277c80bac4ba3ba9764bf0873a04e P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed4b3cc6d19f4a6d9d4b126b8ed6d182 in term 0.
I20260928 16:49:47.149283 11568 leader_election.cc:302] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9567e6c8c4048d38639fbc991c3e211, ed4b3cc6d19f4a6d9d4b126b8ed6d182; no voters:
I20260928 16:49:47.149652 11931 raft_consensus.cc:2781] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:49:47.149799 11931 raft_consensus.cc:482] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:49:47.149902 11931 raft_consensus.cc:3037] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:47.153581 11931 raft_consensus.cc:504] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:47.154490 11931 leader_election.cc:288] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [CANDIDATE]: Term 1 election: Requested vote from peers d9567e6c8c4048d38639fbc991c3e211 (127.9.93.1:38335), f8e2fe45943745edb3479645dd18575c (127.9.93.3:41059)
I20260928 16:49:47.155292 11495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b277c80bac4ba3ba9764bf0873a04e" candidate_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9567e6c8c4048d38639fbc991c3e211"
I20260928 16:49:47.155628 11495 raft_consensus.cc:3037] T 14b277c80bac4ba3ba9764bf0873a04e P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:47.155820 11774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b277c80bac4ba3ba9764bf0873a04e" candidate_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e2fe45943745edb3479645dd18575c"
I20260928 16:49:47.156102 11774 raft_consensus.cc:3037] T 14b277c80bac4ba3ba9764bf0873a04e P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:47.158768 11495 raft_consensus.cc:2454] T 14b277c80bac4ba3ba9764bf0873a04e P d9567e6c8c4048d38639fbc991c3e211 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed4b3cc6d19f4a6d9d4b126b8ed6d182 in term 1.
I20260928 16:49:47.159207 11774 raft_consensus.cc:2454] T 14b277c80bac4ba3ba9764bf0873a04e P f8e2fe45943745edb3479645dd18575c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed4b3cc6d19f4a6d9d4b126b8ed6d182 in term 1.
I20260928 16:49:47.159442 11568 leader_election.cc:302] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9567e6c8c4048d38639fbc991c3e211, ed4b3cc6d19f4a6d9d4b126b8ed6d182; no voters:
I20260928 16:49:47.159818 11931 raft_consensus.cc:2781] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:47.160375 11931 raft_consensus.cc:686] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 1 LEADER]: Becoming Leader. State: Replica: ed4b3cc6d19f4a6d9d4b126b8ed6d182, State: Running, Role: LEADER
I20260928 16:49:47.160773 11931 consensus_queue.cc:237] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:47.164119 11337 catalog_manager.cc:5495] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed4b3cc6d19f4a6d9d4b126b8ed6d182 (127.9.93.2). New cstate: current_term: 1 leader_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } health_report { overall_health: UNKNOWN } } }
I20260928 16:49:47.236688 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:47.231084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60982 (local address 127.9.93.2:41673)
0928 16:49:47.231658 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:47.231663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:47.231681 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:49:47.231753 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:47.231757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:47.231831 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:47.231925 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:47.232309 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:47.233144 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:47.234601 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:47.235074 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:47.235178 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:47.235741 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:47.235885 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:47.235983 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:47.236107 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 16:49:47.236200 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":314,"threads_started":1}
I20260928 16:49:47.247890 11774 raft_consensus.cc:1261] T 14b277c80bac4ba3ba9764bf0873a04e P f8e2fe45943745edb3479645dd18575c [term 1 FOLLOWER]: Refusing update from remote peer ed4b3cc6d19f4a6d9d4b126b8ed6d182: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:49:47.247915 11495 raft_consensus.cc:1261] T 14b277c80bac4ba3ba9764bf0873a04e P d9567e6c8c4048d38639fbc991c3e211 [term 1 FOLLOWER]: Refusing update from remote peer ed4b3cc6d19f4a6d9d4b126b8ed6d182: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:49:47.248668 11954 consensus_queue.cc:1035] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:49:47.249104 11931 consensus_queue.cc:1035] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:49:47.305052 11336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:49:47.305310 11336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:49:47.308772 11336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6: Sending DeleteTablet for 3 replicas of tablet 14b277c80bac4ba3ba9764bf0873a04e
I20260928 16:49:47.309931 11614 tablet_service.cc:1514] Processing DeleteTablet for tablet 14b277c80bac4ba3ba9764bf0873a04e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:49:47 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:60960
I20260928 16:49:47.309948 11755 tablet_service.cc:1514] Processing DeleteTablet for tablet 14b277c80bac4ba3ba9764bf0873a04e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:49:47 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:48150
I20260928 16:49:47.310559 11475 tablet_service.cc:1514] Processing DeleteTablet for tablet 14b277c80bac4ba3ba9764bf0873a04e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:49:47 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:48528
I20260928 16:49:47.310809 11972 tablet_replica.cc:330] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182: stopping tablet replica
I20260928 16:49:47.310887 11973 tablet_replica.cc:330] T 14b277c80bac4ba3ba9764bf0873a04e P f8e2fe45943745edb3479645dd18575c: stopping tablet replica
I20260928 16:49:47.311115 11973 raft_consensus.cc:2229] T 14b277c80bac4ba3ba9764bf0873a04e P f8e2fe45943745edb3479645dd18575c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:49:47.311189 11972 raft_consensus.cc:2229] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:49:47.311021 11336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 16:49:47.311421 11973 raft_consensus.cc:2258] T 14b277c80bac4ba3ba9764bf0873a04e P f8e2fe45943745edb3479645dd18575c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:49:47.311951 11972 raft_consensus.cc:2258] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 16:49:47.312000 11336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:49:47.312840 11973 ts_tablet_manager.cc:1905] T 14b277c80bac4ba3ba9764bf0873a04e P f8e2fe45943745edb3479645dd18575c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:49:47.313223 11972 ts_tablet_manager.cc:1905] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:49:47.319628 11974 tablet_replica.cc:330] T 14b277c80bac4ba3ba9764bf0873a04e P d9567e6c8c4048d38639fbc991c3e211: stopping tablet replica
I20260928 16:49:47.319900 11974 raft_consensus.cc:2229] T 14b277c80bac4ba3ba9764bf0873a04e P d9567e6c8c4048d38639fbc991c3e211 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:49:47.320188 11974 raft_consensus.cc:2258] T 14b277c80bac4ba3ba9764bf0873a04e P d9567e6c8c4048d38639fbc991c3e211 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:49:47.320758 11973 ts_tablet_manager.cc:1918] T 14b277c80bac4ba3ba9764bf0873a04e P f8e2fe45943745edb3479645dd18575c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:49:47.320763 11972 ts_tablet_manager.cc:1918] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:49:47.320921 11972 log.cc:1196] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/14b277c80bac4ba3ba9764bf0873a04e
I20260928 16:49:47.321430 11974 ts_tablet_manager.cc:1905] T 14b277c80bac4ba3ba9764bf0873a04e P d9567e6c8c4048d38639fbc991c3e211: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:49:47.321501 11972 ts_tablet_manager.cc:1939] T 14b277c80bac4ba3ba9764bf0873a04e P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Deleting consensus metadata
I20260928 16:49:47.322469 11614 tablet_service.cc:1467] Processing CreateTablet for tablet 441995fcaf3343c9a01714d7bd5dcbd7 (DEFAULT_TABLE table=test-table [id=bf2bb929709e4894b083a13877d41bbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:49:47.322937 11614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 441995fcaf3343c9a01714d7bd5dcbd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:47.323477 11324 catalog_manager.cc:4841] TS ed4b3cc6d19f4a6d9d4b126b8ed6d182 (127.9.93.2:41673): tablet 14b277c80bac4ba3ba9764bf0873a04e (table test-table [id=4d29a8053413464ca17c4685bdf93969]) successfully deleted
I20260928 16:49:47.324069 11755 tablet_service.cc:1467] Processing CreateTablet for tablet 441995fcaf3343c9a01714d7bd5dcbd7 (DEFAULT_TABLE table=test-table [id=bf2bb929709e4894b083a13877d41bbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:49:47.324075 11475 tablet_service.cc:1467] Processing CreateTablet for tablet 441995fcaf3343c9a01714d7bd5dcbd7 (DEFAULT_TABLE table=test-table [id=bf2bb929709e4894b083a13877d41bbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:49:47.324549 11475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 441995fcaf3343c9a01714d7bd5dcbd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:47.324563 11755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 441995fcaf3343c9a01714d7bd5dcbd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:47.328645 11973 log.cc:1196] T 14b277c80bac4ba3ba9764bf0873a04e P f8e2fe45943745edb3479645dd18575c: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/14b277c80bac4ba3ba9764bf0873a04e
I20260928 16:49:47.329322 11973 ts_tablet_manager.cc:1939] T 14b277c80bac4ba3ba9764bf0873a04e P f8e2fe45943745edb3479645dd18575c: Deleting consensus metadata
I20260928 16:49:47.330686 11324 catalog_manager.cc:4841] TS f8e2fe45943745edb3479645dd18575c (127.9.93.3:41059): tablet 14b277c80bac4ba3ba9764bf0873a04e (table test-table [id=4d29a8053413464ca17c4685bdf93969]) successfully deleted
I20260928 16:49:47.331977 11976 tablet_bootstrap.cc:492] T 441995fcaf3343c9a01714d7bd5dcbd7 P f8e2fe45943745edb3479645dd18575c: Bootstrap starting.
I20260928 16:49:47.336097 11976 tablet_bootstrap.cc:654] T 441995fcaf3343c9a01714d7bd5dcbd7 P f8e2fe45943745edb3479645dd18575c: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:47.337352 11974 ts_tablet_manager.cc:1918] T 14b277c80bac4ba3ba9764bf0873a04e P d9567e6c8c4048d38639fbc991c3e211: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:49:47.337554 11974 log.cc:1196] T 14b277c80bac4ba3ba9764bf0873a04e P d9567e6c8c4048d38639fbc991c3e211: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-0/wal/wals/14b277c80bac4ba3ba9764bf0873a04e
I20260928 16:49:47.337728 11975 tablet_bootstrap.cc:492] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Bootstrap starting.
I20260928 16:49:47.338068 11974 ts_tablet_manager.cc:1939] T 14b277c80bac4ba3ba9764bf0873a04e P d9567e6c8c4048d38639fbc991c3e211: Deleting consensus metadata
I20260928 16:49:47.339567 11324 catalog_manager.cc:4841] TS d9567e6c8c4048d38639fbc991c3e211 (127.9.93.1:38335): tablet 14b277c80bac4ba3ba9764bf0873a04e (table test-table [id=4d29a8053413464ca17c4685bdf93969]) successfully deleted
I20260928 16:49:47.339939 11975 tablet_bootstrap.cc:654] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:47.342228 11977 tablet_bootstrap.cc:492] T 441995fcaf3343c9a01714d7bd5dcbd7 P d9567e6c8c4048d38639fbc991c3e211: Bootstrap starting.
I20260928 16:49:47.342563 11976 tablet_bootstrap.cc:492] T 441995fcaf3343c9a01714d7bd5dcbd7 P f8e2fe45943745edb3479645dd18575c: No bootstrap required, opened a new log
I20260928 16:49:47.342753 11976 ts_tablet_manager.cc:1397] T 441995fcaf3343c9a01714d7bd5dcbd7 P f8e2fe45943745edb3479645dd18575c: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260928 16:49:47.343001 11975 tablet_bootstrap.cc:492] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182: No bootstrap required, opened a new log
I20260928 16:49:47.343211 11975 ts_tablet_manager.cc:1397] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:49:47.343863 11976 raft_consensus.cc:348] T 441995fcaf3343c9a01714d7bd5dcbd7 P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:47.344101 11976 raft_consensus.cc:374] T 441995fcaf3343c9a01714d7bd5dcbd7 P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:47.344197 11977 tablet_bootstrap.cc:654] T 441995fcaf3343c9a01714d7bd5dcbd7 P d9567e6c8c4048d38639fbc991c3e211: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:47.344204 11976 raft_consensus.cc:729] T 441995fcaf3343c9a01714d7bd5dcbd7 P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e2fe45943745edb3479645dd18575c, State: Initialized, Role: FOLLOWER
I20260928 16:49:47.344126 11975 raft_consensus.cc:348] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:47.344424 11975 raft_consensus.cc:374] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:47.344539 11975 raft_consensus.cc:729] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed4b3cc6d19f4a6d9d4b126b8ed6d182, State: Initialized, Role: FOLLOWER
I20260928 16:49:47.344473 11976 consensus_queue.cc:260] T 441995fcaf3343c9a01714d7bd5dcbd7 P f8e2fe45943745edb3479645dd18575c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:47.344817 11975 consensus_queue.cc:260] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:47.345463 11975 ts_tablet_manager.cc:1428] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 16:49:47.346163 11976 ts_tablet_manager.cc:1428] T 441995fcaf3343c9a01714d7bd5dcbd7 P f8e2fe45943745edb3479645dd18575c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:49:47.348807 11977 tablet_bootstrap.cc:492] T 441995fcaf3343c9a01714d7bd5dcbd7 P d9567e6c8c4048d38639fbc991c3e211: No bootstrap required, opened a new log
I20260928 16:49:47.349088 11977 ts_tablet_manager.cc:1397] T 441995fcaf3343c9a01714d7bd5dcbd7 P d9567e6c8c4048d38639fbc991c3e211: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20260928 16:49:47.350167 11977 raft_consensus.cc:348] T 441995fcaf3343c9a01714d7bd5dcbd7 P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:47.350471 11977 raft_consensus.cc:374] T 441995fcaf3343c9a01714d7bd5dcbd7 P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:47.350585 11977 raft_consensus.cc:729] T 441995fcaf3343c9a01714d7bd5dcbd7 P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9567e6c8c4048d38639fbc991c3e211, State: Initialized, Role: FOLLOWER
I20260928 16:49:47.350858 11977 consensus_queue.cc:260] T 441995fcaf3343c9a01714d7bd5dcbd7 P d9567e6c8c4048d38639fbc991c3e211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:47.351392 11977 ts_tablet_manager.cc:1428] T 441995fcaf3343c9a01714d7bd5dcbd7 P d9567e6c8c4048d38639fbc991c3e211: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260928 16:49:47.444561 11931 raft_consensus.cc:482] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:49:47.444854 11931 raft_consensus.cc:504] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:47.445827 11931 leader_election.cc:288] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9567e6c8c4048d38639fbc991c3e211 (127.9.93.1:38335), f8e2fe45943745edb3479645dd18575c (127.9.93.3:41059)
I20260928 16:49:47.446585 11494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441995fcaf3343c9a01714d7bd5dcbd7" candidate_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9567e6c8c4048d38639fbc991c3e211" is_pre_election: true
I20260928 16:49:47.446635 11774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441995fcaf3343c9a01714d7bd5dcbd7" candidate_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e2fe45943745edb3479645dd18575c" is_pre_election: true
I20260928 16:49:47.446897 11774 raft_consensus.cc:2454] T 441995fcaf3343c9a01714d7bd5dcbd7 P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed4b3cc6d19f4a6d9d4b126b8ed6d182 in term 0.
I20260928 16:49:47.446919 11494 raft_consensus.cc:2454] T 441995fcaf3343c9a01714d7bd5dcbd7 P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed4b3cc6d19f4a6d9d4b126b8ed6d182 in term 0.
I20260928 16:49:47.447479 11568 leader_election.cc:302] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9567e6c8c4048d38639fbc991c3e211, ed4b3cc6d19f4a6d9d4b126b8ed6d182; no voters:
I20260928 16:49:47.447825 11931 raft_consensus.cc:2781] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:49:47.447978 11931 raft_consensus.cc:482] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:49:47.448096 11931 raft_consensus.cc:3037] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:47.451602 11931 raft_consensus.cc:504] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:47.452442 11931 leader_election.cc:288] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [CANDIDATE]: Term 1 election: Requested vote from peers d9567e6c8c4048d38639fbc991c3e211 (127.9.93.1:38335), f8e2fe45943745edb3479645dd18575c (127.9.93.3:41059)
I20260928 16:49:47.453222 11774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441995fcaf3343c9a01714d7bd5dcbd7" candidate_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e2fe45943745edb3479645dd18575c"
I20260928 16:49:47.453194 11494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441995fcaf3343c9a01714d7bd5dcbd7" candidate_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9567e6c8c4048d38639fbc991c3e211"
I20260928 16:49:47.453480 11774 raft_consensus.cc:3037] T 441995fcaf3343c9a01714d7bd5dcbd7 P f8e2fe45943745edb3479645dd18575c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:47.453502 11494 raft_consensus.cc:3037] T 441995fcaf3343c9a01714d7bd5dcbd7 P d9567e6c8c4048d38639fbc991c3e211 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:47.456085 11774 raft_consensus.cc:2454] T 441995fcaf3343c9a01714d7bd5dcbd7 P f8e2fe45943745edb3479645dd18575c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed4b3cc6d19f4a6d9d4b126b8ed6d182 in term 1.
I20260928 16:49:47.456676 11568 leader_election.cc:302] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ed4b3cc6d19f4a6d9d4b126b8ed6d182, f8e2fe45943745edb3479645dd18575c; no voters:
I20260928 16:49:47.456975 11494 raft_consensus.cc:2454] T 441995fcaf3343c9a01714d7bd5dcbd7 P d9567e6c8c4048d38639fbc991c3e211 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed4b3cc6d19f4a6d9d4b126b8ed6d182 in term 1.
I20260928 16:49:47.457059 11931 raft_consensus.cc:2781] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:47.457301 11931 raft_consensus.cc:686] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 1 LEADER]: Becoming Leader. State: Replica: ed4b3cc6d19f4a6d9d4b126b8ed6d182, State: Running, Role: LEADER
I20260928 16:49:47.457747 11931 consensus_queue.cc:237] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } }
I20260928 16:49:47.461061 11336 catalog_manager.cc:5495] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed4b3cc6d19f4a6d9d4b126b8ed6d182 (127.9.93.2). New cstate: current_term: 1 leader_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed4b3cc6d19f4a6d9d4b126b8ed6d182" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 } health_report { overall_health: UNKNOWN } } }
I20260928 16:49:47.512243 11494 raft_consensus.cc:1261] T 441995fcaf3343c9a01714d7bd5dcbd7 P d9567e6c8c4048d38639fbc991c3e211 [term 1 FOLLOWER]: Refusing update from remote peer ed4b3cc6d19f4a6d9d4b126b8ed6d182: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:49:47.512261 11774 raft_consensus.cc:1261] T 441995fcaf3343c9a01714d7bd5dcbd7 P f8e2fe45943745edb3479645dd18575c [term 1 FOLLOWER]: Refusing update from remote peer ed4b3cc6d19f4a6d9d4b126b8ed6d182: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:49:47.512822 11971 consensus_queue.cc:1035] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e2fe45943745edb3479645dd18575c" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 41059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:49:47.513227 11931 consensus_queue.cc:1035] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9567e6c8c4048d38639fbc991c3e211" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 16:49:47.536293 11959 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 16:49:49.146811 11337 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:49:49.147090 11337 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:49:49.152809 11337 catalog_manager.cc:5782] T 00000000000000000000000000000000 P facb04e3edf241c2a7d0ceb159d708b6: Sending DeleteTablet for 3 replicas of tablet 441995fcaf3343c9a01714d7bd5dcbd7
I20260928 16:49:49.154066 11475 tablet_service.cc:1514] Processing DeleteTablet for tablet 441995fcaf3343c9a01714d7bd5dcbd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:49:49 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:48528
I20260928 16:49:49.154083 11614 tablet_service.cc:1514] Processing DeleteTablet for tablet 441995fcaf3343c9a01714d7bd5dcbd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:49:49 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:60960
I20260928 16:49:49.155117 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 11408
I20260928 16:49:49.156071 11755 tablet_service.cc:1514] Processing DeleteTablet for tablet 441995fcaf3343c9a01714d7bd5dcbd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:49:49 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:48150
I20260928 16:49:49.159945 12008 tablet_replica.cc:330] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182: stopping tablet replica
I20260928 16:49:49.160224 12008 raft_consensus.cc:2229] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:49:49.160660 12008 raft_consensus.cc:2258] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:49:49.161720 12008 ts_tablet_manager.cc:1905] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:49:49.163913 12009 tablet_replica.cc:330] T 441995fcaf3343c9a01714d7bd5dcbd7 P f8e2fe45943745edb3479645dd18575c: stopping tablet replica
I20260928 16:49:49.165373 12009 raft_consensus.cc:2229] T 441995fcaf3343c9a01714d7bd5dcbd7 P f8e2fe45943745edb3479645dd18575c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:49:49.165804 12009 raft_consensus.cc:2258] T 441995fcaf3343c9a01714d7bd5dcbd7 P f8e2fe45943745edb3479645dd18575c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:49:49.167181 12009 ts_tablet_manager.cc:1905] T 441995fcaf3343c9a01714d7bd5dcbd7 P f8e2fe45943745edb3479645dd18575c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:49:49.169193 12008 ts_tablet_manager.cc:1918] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 16:49:49.169384 12008 log.cc:1196] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/441995fcaf3343c9a01714d7bd5dcbd7
I20260928 16:49:49.169890 12008 ts_tablet_manager.cc:1939] T 441995fcaf3343c9a01714d7bd5dcbd7 P ed4b3cc6d19f4a6d9d4b126b8ed6d182: Deleting consensus metadata
I20260928 16:49:49.169932 12007 tablet_replica.cc:330] T 441995fcaf3343c9a01714d7bd5dcbd7 P d9567e6c8c4048d38639fbc991c3e211: stopping tablet replica
I20260928 16:49:49.170276 12007 raft_consensus.cc:2229] T 441995fcaf3343c9a01714d7bd5dcbd7 P d9567e6c8c4048d38639fbc991c3e211 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:49:49.170634 12007 raft_consensus.cc:2258] T 441995fcaf3343c9a01714d7bd5dcbd7 P d9567e6c8c4048d38639fbc991c3e211 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:49:49.171427 11324 catalog_manager.cc:4841] TS ed4b3cc6d19f4a6d9d4b126b8ed6d182 (127.9.93.2:41673): tablet 441995fcaf3343c9a01714d7bd5dcbd7 (table test-table [id=bf2bb929709e4894b083a13877d41bbf]) successfully deleted
I20260928 16:49:49.172086 12007 ts_tablet_manager.cc:1905] T 441995fcaf3343c9a01714d7bd5dcbd7 P d9567e6c8c4048d38639fbc991c3e211: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:49:49.175871 12009 ts_tablet_manager.cc:1918] T 441995fcaf3343c9a01714d7bd5dcbd7 P f8e2fe45943745edb3479645dd18575c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
Sep 28 16:49:49 dist-test-slave-wzcb krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614186, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.93.1@KRBTEST.COM
I20260928 16:49:49.176060 12009 log.cc:1196] T 441995fcaf3343c9a01714d7bd5dcbd7 P f8e2fe45943745edb3479645dd18575c: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/441995fcaf3343c9a01714d7bd5dcbd7
I20260928 16:49:49.176570 12009 ts_tablet_manager.cc:1939] T 441995fcaf3343c9a01714d7bd5dcbd7 P f8e2fe45943745edb3479645dd18575c: Deleting consensus metadata
I20260928 16:49:49.177811 12007 ts_tablet_manager.cc:1918] T 441995fcaf3343c9a01714d7bd5dcbd7 P d9567e6c8c4048d38639fbc991c3e211: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 16:49:49.177981 12007 log.cc:1196] T 441995fcaf3343c9a01714d7bd5dcbd7 P d9567e6c8c4048d38639fbc991c3e211: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790614166338986-9588-0/minicluster-data/ts-0/wal/wals/441995fcaf3343c9a01714d7bd5dcbd7
I20260928 16:49:49.177958 11324 catalog_manager.cc:4841] TS f8e2fe45943745edb3479645dd18575c (127.9.93.3:41059): tablet 441995fcaf3343c9a01714d7bd5dcbd7 (table test-table [id=bf2bb929709e4894b083a13877d41bbf]) successfully deleted
I20260928 16:49:49.178512 12007 ts_tablet_manager.cc:1939] T 441995fcaf3343c9a01714d7bd5dcbd7 P d9567e6c8c4048d38639fbc991c3e211: Deleting consensus metadata
I20260928 16:49:49.180142 11324 catalog_manager.cc:4841] TS d9567e6c8c4048d38639fbc991c3e211 (127.9.93.1:38335): tablet 441995fcaf3343c9a01714d7bd5dcbd7 (table test-table [id=bf2bb929709e4894b083a13877d41bbf]) successfully deleted
I20260928 16:49:49.184718 12011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:49.158571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48542 (local address 127.9.93.1:38335)
0928 16:49:49.164567 (+ 5996us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:49.164575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:49.164610 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:49:49.164686 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:49.164691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:49.164790 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:49.164974 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:49.167219 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:49.168667 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:49.170807 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:49.171589 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:49.172455 (+ 866us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:49.177428 (+ 4973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:49.177455 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:49.177481 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:49.177530 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:49.181390 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:49.182352 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:49.182362 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:49.182364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:49.182540 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:49.183015 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:49.183020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:49.183022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:49.183567 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:49.183838 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:49.184185 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 16:49:49.184409 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5815,"thread_start_us":5651,"threads_started":1}
W20260928 16:49:49.185567 11540 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48542
W20260928 16:49:49.186288 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 11408: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:49:49.186564 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 11408
I20260928 16:49:49.210381 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 11549
Sep 28 16:49:49 dist-test-slave-wzcb krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614186, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.93.2@KRBTEST.COM
I20260928 16:49:49.228730 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:49.211283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60994 (local address 127.9.93.2:41673)
0928 16:49:49.211869 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:49.211874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:49.211893 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:49:49.211961 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:49.211964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:49.212032 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:49.212131 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:49.213599 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:49.214512 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:49.216084 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:49.216552 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:49.216670 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:49.221734 (+ 5064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:49.221759 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:49.221779 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:49.221843 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:49.225346 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:49.226259 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:49.226268 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:49.226270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:49.226453 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:49.226946 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:49.226950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:49.226952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:49.227522 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:49.227835 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:49.228152 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 16:49:49.228378 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":276,"threads_started":1}
W20260928 16:49:49.229542 11680 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60994
W20260928 16:49:49.230351 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 11549: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:49:49.230548 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 11549
I20260928 16:49:49.251538 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 11689
Sep 28 16:49:49 dist-test-slave-wzcb krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614186, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.93.3@KRBTEST.COM
I20260928 16:49:49.270066 12013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:49.252401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48156 (local address 127.9.93.3:41059)
0928 16:49:49.253066 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:49.253071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:49.253089 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:49:49.253157 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:49.253160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:49.253228 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:49.253321 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:49.254992 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:49.255952 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:49.257338 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:49.257805 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:49.257918 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:49.262785 (+ 4867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:49.262807 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:49.262827 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:49.262903 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:49.266413 (+ 3510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:49.267422 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:49.267432 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:49.267435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:49.267642 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:49.268142 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:49.268147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:49.268149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:49.268655 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:49.268915 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:49.269362 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 16:49:49.269667 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":341,"threads_started":1}
W20260928 16:49:49.270910 11820 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48156
W20260928 16:49:49.271629 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 11689: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:49:49.271817 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 11689
I20260928 16:49:49.287663 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 11305
I20260928 16:49:49.289044 11377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:49:49.411283 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 11305
2026-09-28T16:49:49Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9660 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:49:49 dist-test-slave-wzcb krb5kdc[12017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:49:49 dist-test-slave-wzcb krb5kdc[12017](info): set up 2 sockets
Sep 28 16:49:49 dist-test-slave-wzcb krb5kdc[12017](info): commencing operation
krb5kdc: starting...
W20260928 16:49:50.426667 9588 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:49:50 dist-test-slave-wzcb krb5kdc[12017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:49:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:49:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:49:50.640123 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:37987
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:39245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:37987
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:50.996320 12030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:50.996690 12030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:50.996821 12030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:51.005054 12030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:51.005205 12030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:51.005260 12030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:51.005345 12030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:51.005398 12030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:51.016178 12030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:37987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:37987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:51.017760 12030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:51.019284 12030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:51.032810 12035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:51.032737 12038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:51.032783 12036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:51.033771 12030 server_base.cc:1034] running on GCE node
I20260928 16:49:51.034749 12030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:51.036535 12030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:51.037784 12030 hybrid_clock.cc:648] HybridClock initialized: now 1790614191037713 us; error 67 us; skew 500 ppm
I20260928 16:49:51.038194 12030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:51 dist-test-slave-wzcb krb5kdc[12017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614191, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:51.041898 12030 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:49:51.043661 12030 webserver.cc:460] Webserver started at http://127.9.93.62:39693/ using document root <none> and password file <none>
I20260928 16:49:51.044469 12030 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:51.044616 12030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:51.044967 12030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:51.048506 12030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "f7a862c722dc446c8610851d11050789"
format_stamp: "Formatted at 2026-09-28 16:49:51 on dist-test-slave-wzcb"
server_key: "14622da22d73c2b736059793f8895277"
server_key_iv: "b00c9a40194748d170b9503056266f86"
server_key_version: "encryptionkey@0"
I20260928 16:49:51.049420 12030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "f7a862c722dc446c8610851d11050789"
format_stamp: "Formatted at 2026-09-28 16:49:51 on dist-test-slave-wzcb"
server_key: "14622da22d73c2b736059793f8895277"
server_key_iv: "b00c9a40194748d170b9503056266f86"
server_key_version: "encryptionkey@0"
I20260928 16:49:51.055964 12030 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 16:49:51.060638 12045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:51.062595 12030 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:49:51.062832 12030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "f7a862c722dc446c8610851d11050789"
format_stamp: "Formatted at 2026-09-28 16:49:51 on dist-test-slave-wzcb"
server_key: "14622da22d73c2b736059793f8895277"
server_key_iv: "b00c9a40194748d170b9503056266f86"
server_key_version: "encryptionkey@0"
I20260928 16:49:51.063072 12030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:51.085227 12030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:51.090298 12030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:51.090621 12030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:51.109982 12030 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:37987
I20260928 16:49:51.110016 12096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:37987 every 8 connection(s)
I20260928 16:49:51.111699 12030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:49:51.112444 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 12030
I20260928 16:49:51.112680 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:49:51.113473 9588 external_mini_cluster.cc:1428] Setting key 3e4807880759e89d1c2fbdb9d2a3785d
I20260928 16:49:51.116750 12097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 16:49:51 dist-test-slave-wzcb krb5kdc[12017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:49:51.131783 12097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7a862c722dc446c8610851d11050789: Bootstrap starting.
I20260928 16:49:51.136404 12097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7a862c722dc446c8610851d11050789: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:51.137820 12097 log.cc:824] T 00000000000000000000000000000000 P f7a862c722dc446c8610851d11050789: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:51.140332 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:51.115775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39982 (local address 127.9.93.62:37987)
0928 16:49:51.116670 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:51.116691 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:51.116756 (+ 65us) server_negotiation.cc:407] Connection header received
0928 16:49:51.118840 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:51.118871 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:51.119370 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:51.119777 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:51.121770 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:51.123288 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:51.125215 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:51.126006 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:51.126180 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:51.131188 (+ 5008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:51.131202 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:51.131228 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:51.131291 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:51.135404 (+ 4113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:51.136237 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:51.136248 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:51.136258 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:51.136494 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:51.137042 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:51.137046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:51.137049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:51.137686 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:51.137917 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:51.138533 (+ 616us) server_negotiation.cc:299] Negotiation successful
0928 16:49:51.139110 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":381,"threads_started":1}
I20260928 16:49:51.141499 12097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7a862c722dc446c8610851d11050789: No bootstrap required, opened a new log
I20260928 16:49:51.150286 12097 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7a862c722dc446c8610851d11050789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a862c722dc446c8610851d11050789" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 37987 } }
I20260928 16:49:51.150738 12097 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7a862c722dc446c8610851d11050789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:51.150852 12097 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7a862c722dc446c8610851d11050789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7a862c722dc446c8610851d11050789, State: Initialized, Role: FOLLOWER
I20260928 16:49:51.151521 12097 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7a862c722dc446c8610851d11050789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a862c722dc446c8610851d11050789" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 37987 } }
I20260928 16:49:51.151750 12097 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7a862c722dc446c8610851d11050789 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:49:51.151891 12097 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7a862c722dc446c8610851d11050789 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:49:51.152067 12097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7a862c722dc446c8610851d11050789 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:51.154518 12097 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7a862c722dc446c8610851d11050789 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a862c722dc446c8610851d11050789" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 37987 } }
I20260928 16:49:51.155149 12097 leader_election.cc:302] T 00000000000000000000000000000000 P f7a862c722dc446c8610851d11050789 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7a862c722dc446c8610851d11050789; no voters:
I20260928 16:49:51.155916 12097 leader_election.cc:288] T 00000000000000000000000000000000 P f7a862c722dc446c8610851d11050789 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:49:51.156078 12102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7a862c722dc446c8610851d11050789 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:51.156981 12102 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7a862c722dc446c8610851d11050789 [term 1 LEADER]: Becoming Leader. State: Replica: f7a862c722dc446c8610851d11050789, State: Running, Role: LEADER
I20260928 16:49:51.157599 12102 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7a862c722dc446c8610851d11050789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a862c722dc446c8610851d11050789" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 37987 } }
I20260928 16:49:51.158270 12097 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7a862c722dc446c8610851d11050789 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:51.160626 12103 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7a862c722dc446c8610851d11050789 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7a862c722dc446c8610851d11050789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a862c722dc446c8610851d11050789" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 37987 } } }
I20260928 16:49:51.161142 12103 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7a862c722dc446c8610851d11050789 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:51.161046 12104 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7a862c722dc446c8610851d11050789 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7a862c722dc446c8610851d11050789. Latest consensus state: current_term: 1 leader_uuid: "f7a862c722dc446c8610851d11050789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a862c722dc446c8610851d11050789" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 37987 } } }
I20260928 16:49:51.161376 12104 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7a862c722dc446c8610851d11050789 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:51.166374 12111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:49:51.172065 12111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:49:51.182371 12111 catalog_manager.cc:1347] Generated new cluster ID: b3da74180de54f1080bba216d91a6252
I20260928 16:49:51.182512 12111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:49:51.196310 12111 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:49:51.198360 12111 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:49:51.212872 12111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7a862c722dc446c8610851d11050789: Generated new TSK 0
I20260928 16:49:51.213948 12111 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
I20260928 16:49:51.272641 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:37987
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:39245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:51.647867 12123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:51.648291 12123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:51.648460 12123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:51.656850 12123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:51.657032 12123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:51.657208 12123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:49:51.669159 12123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:37987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:51.670748 12123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:51.672327 12123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:51.688509 12129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:51.688390 12128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:51.688429 12131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:51.693427 12123 server_base.cc:1034] running on GCE node
I20260928 16:49:51.694334 12123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:51.695657 12123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:51.696903 12123 hybrid_clock.cc:648] HybridClock initialized: now 1790614191696808 us; error 87 us; skew 500 ppm
I20260928 16:49:51.697297 12123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:51 dist-test-slave-wzcb krb5kdc[12017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614191, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:51.701524 12123 init.cc:377] Logged in from keytab as kudu/127.9.93.1@KRBTEST.COM (short username kudu)
I20260928 16:49:51.703675 12123 webserver.cc:460] Webserver started at http://127.9.93.1:36901/ using document root <none> and password file <none>
I20260928 16:49:51.704600 12123 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:51.704756 12123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:51.705134 12123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:51.708988 12123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "c745b57e83b749e4a505a0396da7a642"
format_stamp: "Formatted at 2026-09-28 16:49:51 on dist-test-slave-wzcb"
server_key: "1816fd48bd902d1f757ba4df9f55585b"
server_key_iv: "b7df3a298d3436ce588a1334aad06b70"
server_key_version: "encryptionkey@0"
I20260928 16:49:51.710042 12123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "c745b57e83b749e4a505a0396da7a642"
format_stamp: "Formatted at 2026-09-28 16:49:51 on dist-test-slave-wzcb"
server_key: "1816fd48bd902d1f757ba4df9f55585b"
server_key_iv: "b7df3a298d3436ce588a1334aad06b70"
server_key_version: "encryptionkey@0"
I20260928 16:49:51.716728 12123 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 16:49:51.721608 12138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:51.723471 12123 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:49:51.723701 12123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "c745b57e83b749e4a505a0396da7a642"
format_stamp: "Formatted at 2026-09-28 16:49:51 on dist-test-slave-wzcb"
server_key: "1816fd48bd902d1f757ba4df9f55585b"
server_key_iv: "b7df3a298d3436ce588a1334aad06b70"
server_key_version: "encryptionkey@0"
I20260928 16:49:51.723961 12123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:51.743089 12123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:51.748241 12123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:51.748593 12123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:51.749766 12123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:49:51.751533 12123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:49:51.751624 12123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:51.751773 12123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:49:51.751832 12123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:51.797452 12123 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:46809
I20260928 16:49:51.797514 12250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:46809 every 8 connection(s)
I20260928 16:49:51.798945 12123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:49:51.801469 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 12123
I20260928 16:49:51.801788 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:49:51.802670 9588 external_mini_cluster.cc:1428] Setting key 323cd76297ba07355f518ef5b57f7271
Sep 28 16:49:51 dist-test-slave-wzcb krb5kdc[12017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614191, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.2@KRBTEST.COM" created.
I20260928 16:49:51.828704 12253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:51.802417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:35789 (local address 127.9.93.62:37987)
0928 16:49:51.802964 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:51.802971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:51.804953 (+ 1982us) server_negotiation.cc:407] Connection header received
0928 16:49:51.807630 (+ 2677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:51.807637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:51.807733 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:51.807856 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:51.810414 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:51.811517 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:51.813507 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:51.814218 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:51.814374 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:51.819667 (+ 5293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:51.819676 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:51.819682 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:51.819739 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:51.823593 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:51.824972 (+ 1379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:51.824978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:51.824981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:51.825174 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:51.825885 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:51.825890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:51.825892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:51.826235 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:51.826374 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:51.827883 (+ 1509us) server_negotiation.cc:299] Negotiation successful
0928 16:49:51.828293 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"spinlock_wait_cycles":43264,"thread_start_us":356,"threads_started":1}
I20260928 16:49:51.830493 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:51.803228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:37987 (local address 127.9.93.1:35789)
0928 16:49:51.804320 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:51.804379 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:51.807179 (+ 2800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:51.808222 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:51.808236 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:51.809151 (+ 915us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:51.810160 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:51.810193 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:51.811716 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:51.811720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:51.813289 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:51.813306 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:51.814518 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:51.814524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:51.814867 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:51.815806 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:51.815838 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:51.819258 (+ 3420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:51.823882 (+ 4624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:51.823898 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:51.823925 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:51.824779 (+ 854us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:51.825309 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:51.825315 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:51.825319 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:51.825700 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:51.826384 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:51.826395 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:51.827127 (+ 732us) client_negotiation.cc:769] Sending connection context
0928 16:49:51.828028 (+ 901us) client_negotiation.cc:240] Negotiation successful
0928 16:49:51.828812 (+ 784us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":515,"threads_started":1}
I20260928 16:49:51.832882 12251 heartbeater.cc:347] Connected to a master server at 127.9.93.62:37987
I20260928 16:49:51.833307 12251 heartbeater.cc:464] Registering TS with master...
I20260928 16:49:51.834380 12251 heartbeater.cc:511] Master 127.9.93.62:37987 requested a full tablet report, sending...
I20260928 16:49:51.837378 12062 ts_manager.cc:194] Registered new tserver with Master: c745b57e83b749e4a505a0396da7a642 (127.9.93.1:46809)
I20260928 16:49:51.839711 12062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.1@KRBTEST.COM'} at 127.9.93.1:35789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
I20260928 16:49:51.858990 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:37987
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:39245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:52.241016 12257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:52.241380 12257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:52.241544 12257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:52.249887 12257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:52.250072 12257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:52.250249 12257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:49:52.262348 12257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:37987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:52.264056 12257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:52.265666 12257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:52.281207 12262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:52.281384 12265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:52.281296 12263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:52.282658 12257 server_base.cc:1034] running on GCE node
I20260928 16:49:52.283500 12257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:52.284680 12257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:52.285923 12257 hybrid_clock.cc:648] HybridClock initialized: now 1790614192285830 us; error 78 us; skew 500 ppm
I20260928 16:49:52.286320 12257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:52 dist-test-slave-wzcb krb5kdc[12017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614192, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:52.290162 12257 init.cc:377] Logged in from keytab as kudu/127.9.93.2@KRBTEST.COM (short username kudu)
I20260928 16:49:52.292184 12257 webserver.cc:460] Webserver started at http://127.9.93.2:45503/ using document root <none> and password file <none>
I20260928 16:49:52.293028 12257 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:52.293164 12257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:52.293530 12257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:52.297123 12257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "405124e8a726474cadd840bda2fc38ab"
format_stamp: "Formatted at 2026-09-28 16:49:52 on dist-test-slave-wzcb"
server_key: "b0dd79e41854b91d3bb998267842e8ae"
server_key_iv: "1b4418fc77eee71e53c3ee2c5fe25c9e"
server_key_version: "encryptionkey@0"
I20260928 16:49:52.298151 12257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "405124e8a726474cadd840bda2fc38ab"
format_stamp: "Formatted at 2026-09-28 16:49:52 on dist-test-slave-wzcb"
server_key: "b0dd79e41854b91d3bb998267842e8ae"
server_key_iv: "1b4418fc77eee71e53c3ee2c5fe25c9e"
server_key_version: "encryptionkey@0"
I20260928 16:49:52.305001 12257 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 16:49:52.309556 12272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:52.311269 12257 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:49:52.311555 12257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "405124e8a726474cadd840bda2fc38ab"
format_stamp: "Formatted at 2026-09-28 16:49:52 on dist-test-slave-wzcb"
server_key: "b0dd79e41854b91d3bb998267842e8ae"
server_key_iv: "1b4418fc77eee71e53c3ee2c5fe25c9e"
server_key_version: "encryptionkey@0"
I20260928 16:49:52.311810 12257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:52.335776 12257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:52.341400 12257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:52.341759 12257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:52.342953 12257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:49:52.344726 12257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:49:52.344820 12257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:52.344934 12257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:49:52.344993 12257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:52.380606 12257 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:34723
I20260928 16:49:52.380681 12385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:34723 every 8 connection(s)
I20260928 16:49:52.382149 12257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:49:52.383108 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 12257
I20260928 16:49:52.383360 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:49:52.383862 9588 external_mini_cluster.cc:1428] Setting key 9af753ce327e93371193b20c5268c284
Sep 28 16:49:52 dist-test-slave-wzcb krb5kdc[12017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614192, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.3@KRBTEST.COM" created.
I20260928 16:49:52.408870 12388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:52.385500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:58547 (local address 127.9.93.62:37987)
0928 16:49:52.386052 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:52.386057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:52.387593 (+ 1536us) server_negotiation.cc:407] Connection header received
0928 16:49:52.389755 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:52.389761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:52.389853 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:52.389983 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:52.392061 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:52.392909 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:52.395669 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:52.396164 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:52.396281 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:52.401485 (+ 5204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:52.401491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:52.401495 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:52.401540 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:52.404460 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:52.405795 (+ 1335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:52.405801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:52.405811 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:52.406004 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:52.406615 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:52.406620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:52.406622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:52.406973 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:52.407109 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:52.408177 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0928 16:49:52.408476 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":293,"threads_started":1}
I20260928 16:49:52.410040 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:52.386374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:37987 (local address 127.9.93.2:58547)
0928 16:49:52.387386 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:52.387441 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:52.389426 (+ 1985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:52.390281 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:52.390291 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:52.390991 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:52.391815 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:52.391847 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:52.393140 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:52.393148 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:52.395419 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:52.395452 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:52.396472 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:52.396479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:52.396827 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:52.397685 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:52.397719 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:52.401030 (+ 3311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:52.404709 (+ 3679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:52.404724 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:52.404749 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:52.405579 (+ 830us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:52.406151 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:52.406156 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:52.406160 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:52.406419 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:52.407120 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:52.407128 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:52.407705 (+ 577us) client_negotiation.cc:769] Sending connection context
0928 16:49:52.407989 (+ 284us) client_negotiation.cc:240] Negotiation successful
0928 16:49:52.408662 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":363,"threads_started":1}
I20260928 16:49:52.412042 12386 heartbeater.cc:347] Connected to a master server at 127.9.93.62:37987
I20260928 16:49:52.412494 12386 heartbeater.cc:464] Registering TS with master...
I20260928 16:49:52.413632 12386 heartbeater.cc:511] Master 127.9.93.62:37987 requested a full tablet report, sending...
I20260928 16:49:52.415861 12062 ts_manager.cc:194] Registered new tserver with Master: 405124e8a726474cadd840bda2fc38ab (127.9.93.2:34723)
I20260928 16:49:52.417101 12062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.2@KRBTEST.COM'} at 127.9.93.2:58547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
I20260928 16:49:52.435078 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:37987
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:39245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:52.812252 12392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:52.812618 12392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:52.812772 12392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:52.821054 12392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:52.821230 12392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:52.821413 12392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:49:52.832962 12392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:37987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:52.834646 12392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:52.836611 12392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:49:52.844693 12251 heartbeater.cc:503] Master 127.9.93.62:37987 was elected leader, sending a full tablet report...
W20260928 16:49:52.854588 12397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:52.854619 12398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:52.854662 12400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:52.855410 12392 server_base.cc:1034] running on GCE node
I20260928 16:49:52.856369 12392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:52.857837 12392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:52.859108 12392 hybrid_clock.cc:648] HybridClock initialized: now 1790614192859046 us; error 51 us; skew 500 ppm
I20260928 16:49:52.859556 12392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:52 dist-test-slave-wzcb krb5kdc[12017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614192, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:52.869794 12392 init.cc:377] Logged in from keytab as kudu/127.9.93.3@KRBTEST.COM (short username kudu)
I20260928 16:49:52.871901 12392 webserver.cc:460] Webserver started at http://127.9.93.3:35899/ using document root <none> and password file <none>
I20260928 16:49:52.872792 12392 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:52.872933 12392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:52.873323 12392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:52.877146 12392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "f3e831fae84841a18cfd86d85bd83fef"
format_stamp: "Formatted at 2026-09-28 16:49:52 on dist-test-slave-wzcb"
server_key: "908ccc4e2343eb1801302df595e8bc79"
server_key_iv: "ea41e2b07216c91efd524647ef3a3fb2"
server_key_version: "encryptionkey@0"
I20260928 16:49:52.878237 12392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "f3e831fae84841a18cfd86d85bd83fef"
format_stamp: "Formatted at 2026-09-28 16:49:52 on dist-test-slave-wzcb"
server_key: "908ccc4e2343eb1801302df595e8bc79"
server_key_iv: "ea41e2b07216c91efd524647ef3a3fb2"
server_key_version: "encryptionkey@0"
I20260928 16:49:52.885023 12392 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 16:49:52.889393 12407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:52.891260 12392 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:49:52.891574 12392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "f3e831fae84841a18cfd86d85bd83fef"
format_stamp: "Formatted at 2026-09-28 16:49:52 on dist-test-slave-wzcb"
server_key: "908ccc4e2343eb1801302df595e8bc79"
server_key_iv: "ea41e2b07216c91efd524647ef3a3fb2"
server_key_version: "encryptionkey@0"
I20260928 16:49:52.891841 12392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:52.919226 12392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:52.924358 12392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:52.924690 12392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:52.925855 12392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:49:52.927580 12392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:49:52.927670 12392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:52.927804 12392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:49:52.927860 12392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:52.962325 12392 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:38289
I20260928 16:49:52.962397 12519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:38289 every 8 connection(s)
I20260928 16:49:52.963862 12392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:49:52.968713 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 12392
I20260928 16:49:52.968948 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:49:52.969522 9588 external_mini_cluster.cc:1428] Setting key baa6e6640969c1322b1a07dfbfc29653
Sep 28 16:49:52 dist-test-slave-wzcb krb5kdc[12017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614192, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:49:52.988757 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:52.966796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:33505 (local address 127.9.93.62:37987)
0928 16:49:52.967329 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:52.967337 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:52.968804 (+ 1467us) server_negotiation.cc:407] Connection header received
0928 16:49:52.971329 (+ 2525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:52.971334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:52.971432 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:52.971532 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:52.973456 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:52.974305 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:52.976274 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:52.976779 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:52.976904 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:52.981335 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:52.981343 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:52.981346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:52.981402 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:52.984611 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:52.985876 (+ 1265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:52.985883 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:52.985887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:52.986111 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:52.986760 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:52.986763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:52.986766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:52.987120 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:52.987265 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:52.988209 (+ 944us) server_negotiation.cc:299] Negotiation successful
0928 16:49:52.988450 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":266,"threads_started":1}
I20260928 16:49:52.989877 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:52.967621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:37987 (local address 127.9.93.3:33505)
0928 16:49:52.968578 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:52.968650 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:52.970970 (+ 2320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:52.971812 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:52.971823 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:52.972476 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:52.973276 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:52.973301 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:52.974479 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:52.974491 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:52.976066 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:52.976082 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:52.977065 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:52.977071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:52.977332 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:52.978151 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:52.978179 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:52.981011 (+ 2832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:52.984811 (+ 3800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:52.984824 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:52.984848 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:52.985683 (+ 835us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:52.986276 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:52.986280 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:52.986283 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:52.986619 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:52.987273 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:52.987281 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:52.987760 (+ 479us) client_negotiation.cc:769] Sending connection context
0928 16:49:52.988038 (+ 278us) client_negotiation.cc:240] Negotiation successful
0928 16:49:52.988564 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":303,"threads_started":1}
I20260928 16:49:52.991585 12520 heartbeater.cc:347] Connected to a master server at 127.9.93.62:37987
I20260928 16:49:52.991979 12520 heartbeater.cc:464] Registering TS with master...
I20260928 16:49:52.992899 12520 heartbeater.cc:511] Master 127.9.93.62:37987 requested a full tablet report, sending...
I20260928 16:49:52.994720 12062 ts_manager.cc:194] Registered new tserver with Master: f3e831fae84841a18cfd86d85bd83fef (127.9.93.3:38289)
I20260928 16:49:52.995692 12062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.3@KRBTEST.COM'} at 127.9.93.3:33505
I20260928 16:49:52.999363 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 16:49:53.036250 12522 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:53.033450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34288 (local address 127.9.93.62:37987)
0928 16:49:53.033658 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:53.033663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:53.034151 (+ 488us) server_negotiation.cc:407] Connection header received
0928 16:49:53.034333 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:53.034338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:53.034479 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:53.034587 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:53.036031 (+ 1444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34288: BlockingRecv error: recv got EOF from 127.0.0.1:34288 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 16:49:53.039458 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 12123
W20260928 16:49:53.042977 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 12123: Not authorized: Client connection negotiation failed: client connection to 127.9.93.1:46809: server requires authentication, but client does not have Kerberos credentials available
I20260928 16:49:53.043221 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 12123
I20260928 16:49:53.057984 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 12257
W20260928 16:49:53.061257 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 12257: Not authorized: Client connection negotiation failed: client connection to 127.9.93.2:34723: server requires authentication, but client does not have Kerberos credentials available
I20260928 16:49:53.061480 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 12257
I20260928 16:49:53.076112 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 12392
W20260928 16:49:53.079555 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 12392: Not authorized: Client connection negotiation failed: client connection to 127.9.93.3:38289: server requires authentication, but client does not have Kerberos credentials available
I20260928 16:49:53.079736 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 12392
I20260928 16:49:53.093427 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 12030
I20260928 16:49:53.094817 12092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:49:53.204038 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 12030
2026-09-28T16:49:53Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3794 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:49:53 dist-test-slave-wzcb krb5kdc[12538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:49:53 dist-test-slave-wzcb krb5kdc[12538](info): set up 2 sockets
Sep 28 16:49:53 dist-test-slave-wzcb krb5kdc[12538](info): commencing operation
krb5kdc: starting...
W20260928 16:49:54.230233 9588 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:49:54 dist-test-slave-wzcb krb5kdc[12538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:49:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:49:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:49:54.447692 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:44175
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:39705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:44175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:54.806169 12551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:54.806568 12551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:54.806752 12551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:54.814435 12551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:54.814574 12551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:54.814621 12551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:54.814682 12551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:54.814733 12551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:54.825613 12551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:44175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:44175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:54.827137 12551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:54.828788 12551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:54.842535 12556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:54.842521 12557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:54.842486 12559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:54.843600 12551 server_base.cc:1034] running on GCE node
I20260928 16:49:54.844568 12551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:54.846333 12551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:54.847563 12551 hybrid_clock.cc:648] HybridClock initialized: now 1790614194847503 us; error 51 us; skew 500 ppm
I20260928 16:49:54.847985 12551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:54 dist-test-slave-wzcb krb5kdc[12538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614194, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:54.851482 12551 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:49:54.853487 12551 webserver.cc:460] Webserver started at http://127.9.93.62:40539/ using document root <none> and password file <none>
I20260928 16:49:54.854350 12551 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:54.854501 12551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:54.854852 12551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:54.858528 12551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "4e8a9bd3667d45ac8e36a8f99c73df70"
format_stamp: "Formatted at 2026-09-28 16:49:54 on dist-test-slave-wzcb"
server_key: "e1e7095df4f3dab1be3bd6014c80daba"
server_key_iv: "db4102ecb8ae273e81d5336fe2fbe145"
server_key_version: "encryptionkey@0"
I20260928 16:49:54.859566 12551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "4e8a9bd3667d45ac8e36a8f99c73df70"
format_stamp: "Formatted at 2026-09-28 16:49:54 on dist-test-slave-wzcb"
server_key: "e1e7095df4f3dab1be3bd6014c80daba"
server_key_iv: "db4102ecb8ae273e81d5336fe2fbe145"
server_key_version: "encryptionkey@0"
I20260928 16:49:54.866240 12551 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:49:54.870990 12566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:54.872923 12551 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:49:54.873159 12551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "4e8a9bd3667d45ac8e36a8f99c73df70"
format_stamp: "Formatted at 2026-09-28 16:49:54 on dist-test-slave-wzcb"
server_key: "e1e7095df4f3dab1be3bd6014c80daba"
server_key_iv: "db4102ecb8ae273e81d5336fe2fbe145"
server_key_version: "encryptionkey@0"
I20260928 16:49:54.873399 12551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:54.896392 12551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:54.901427 12551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:54.901762 12551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:54.921531 12551 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:44175
I20260928 16:49:54.921604 12617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:44175 every 8 connection(s)
I20260928 16:49:54.923098 12551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:49:54.927897 12618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:54.930372 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 12551
I20260928 16:49:54.930637 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:49:54.931603 9588 external_mini_cluster.cc:1428] Setting key cbcd2377ded9f09b9411fc2b66aaf090
I20260928 16:49:54.943188 12618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e8a9bd3667d45ac8e36a8f99c73df70: Bootstrap starting.
I20260928 16:49:54.947939 12618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e8a9bd3667d45ac8e36a8f99c73df70: Neither blocks nor log segments found. Creating new log.
Sep 28 16:49:54 dist-test-slave-wzcb krb5kdc[12538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:49:54.949265 12618 log.cc:824] T 00000000000000000000000000000000 P 4e8a9bd3667d45ac8e36a8f99c73df70: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:54.953044 12618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e8a9bd3667d45ac8e36a8f99c73df70: No bootstrap required, opened a new log
I20260928 16:49:54.958163 12621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:54.934016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37684 (local address 127.9.93.62:44175)
0928 16:49:54.934986 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:54.935003 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:54.935059 (+ 56us) server_negotiation.cc:407] Connection header received
0928 16:49:54.936913 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:54.936941 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:54.937451 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:54.937849 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:54.940102 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:54.941698 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:54.943667 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:54.944474 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:54.944643 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:54.949661 (+ 5018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:54.949672 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:54.949690 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:54.949742 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:54.953118 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:54.954168 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:54.954175 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:54.954182 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:54.954376 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:54.954842 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:54.954845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:54.954847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:54.955588 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:54.955846 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:54.956393 (+ 547us) server_negotiation.cc:299] Negotiation successful
0928 16:49:54.956891 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":410,"threads_started":1}
I20260928 16:49:54.963229 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:49:54.963143 12618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e8a9bd3667d45ac8e36a8f99c73df70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8a9bd3667d45ac8e36a8f99c73df70" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 44175 } }
I20260928 16:49:54.963683 12618 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e8a9bd3667d45ac8e36a8f99c73df70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:54.963788 12618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e8a9bd3667d45ac8e36a8f99c73df70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e8a9bd3667d45ac8e36a8f99c73df70, State: Initialized, Role: FOLLOWER
I20260928 16:49:54.964434 12618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e8a9bd3667d45ac8e36a8f99c73df70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8a9bd3667d45ac8e36a8f99c73df70" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 44175 } }
I20260928 16:49:54.964695 12618 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e8a9bd3667d45ac8e36a8f99c73df70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:49:54.964834 12618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e8a9bd3667d45ac8e36a8f99c73df70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:49:54.965010 12618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e8a9bd3667d45ac8e36a8f99c73df70 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:54.967489 12618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e8a9bd3667d45ac8e36a8f99c73df70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8a9bd3667d45ac8e36a8f99c73df70" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 44175 } }
I20260928 16:49:54.968127 12618 leader_election.cc:302] T 00000000000000000000000000000000 P 4e8a9bd3667d45ac8e36a8f99c73df70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e8a9bd3667d45ac8e36a8f99c73df70; no voters:
I20260928 16:49:54.968792 12618 leader_election.cc:288] T 00000000000000000000000000000000 P 4e8a9bd3667d45ac8e36a8f99c73df70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:49:54.968969 12623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e8a9bd3667d45ac8e36a8f99c73df70 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:54.970135 12623 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e8a9bd3667d45ac8e36a8f99c73df70 [term 1 LEADER]: Becoming Leader. State: Replica: 4e8a9bd3667d45ac8e36a8f99c73df70, State: Running, Role: LEADER
I20260928 16:49:54.970806 12623 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e8a9bd3667d45ac8e36a8f99c73df70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8a9bd3667d45ac8e36a8f99c73df70" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 44175 } }
I20260928 16:49:54.971400 12618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e8a9bd3667d45ac8e36a8f99c73df70 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:54.974334 12625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e8a9bd3667d45ac8e36a8f99c73df70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e8a9bd3667d45ac8e36a8f99c73df70. Latest consensus state: current_term: 1 leader_uuid: "4e8a9bd3667d45ac8e36a8f99c73df70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8a9bd3667d45ac8e36a8f99c73df70" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 44175 } } }
I20260928 16:49:54.974768 12625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e8a9bd3667d45ac8e36a8f99c73df70 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:54.975535 12624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e8a9bd3667d45ac8e36a8f99c73df70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e8a9bd3667d45ac8e36a8f99c73df70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8a9bd3667d45ac8e36a8f99c73df70" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 44175 } } }
I20260928 16:49:54.975909 12624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e8a9bd3667d45ac8e36a8f99c73df70 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:54.976120 12630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:49:54.989904 12630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:49:55.013233 12630 catalog_manager.cc:1347] Generated new cluster ID: f08c9f0a9be74a0385cf0ab249cd623a
I20260928 16:49:55.013388 12630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:49:55.033116 12630 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:49:55.035372 12630 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:49:55.049651 12630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e8a9bd3667d45ac8e36a8f99c73df70: Generated new TSK 0
I20260928 16:49:55.051014 12630 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:49:55.063225 12621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:55.052408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37686 (local address 127.9.93.62:44175)
0928 16:49:55.052610 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:55.052618 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:55.054161 (+ 1543us) server_negotiation.cc:407] Connection header received
0928 16:49:55.054497 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:55.054503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:55.054605 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:55.054694 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:55.056370 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:55.057225 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:55.059277 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:55.059860 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:55.060021 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:55.061403 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:55.061411 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:55.061415 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:49:55.062256 (+ 841us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:55.062336 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:49:55.062374 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:55.062507 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:55.062730 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 16:49:55.062855 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:49:55.073295 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 12551
I20260928 16:49:55.074607 12613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:49:55.170332 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 12551
2026-09-28T16:49:55Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1957 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:49:55 dist-test-slave-wzcb krb5kdc[12653](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:49:55 dist-test-slave-wzcb krb5kdc[12653](info): set up 2 sockets
Sep 28 16:49:55 dist-test-slave-wzcb krb5kdc[12653](info): commencing operation
krb5kdc: starting...
W20260928 16:49:56.186164 9588 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:49:56 dist-test-slave-wzcb krb5kdc[12653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:49:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:49:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:49:56.403779 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:37697
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:44531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:37697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:56.757102 12666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:56.757443 12666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:56.757539 12666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:56.765204 12666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:56.765348 12666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:56.765395 12666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:56.765455 12666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:56.765501 12666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:56.776110 12666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:44531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:37697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:37697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:56.777671 12666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:56.779196 12666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:56.792356 12672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:56.792380 12671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:56.793139 12674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:56.793202 12666 server_base.cc:1034] running on GCE node
I20260928 16:49:56.794181 12666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:56.795991 12666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:56.797230 12666 hybrid_clock.cc:648] HybridClock initialized: now 1790614196797168 us; error 56 us; skew 500 ppm
I20260928 16:49:56.797627 12666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:56 dist-test-slave-wzcb krb5kdc[12653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614196, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:56.801079 12666 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:49:56.802836 12666 webserver.cc:460] Webserver started at http://127.9.93.62:43633/ using document root <none> and password file <none>
I20260928 16:49:56.803717 12666 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:56.803860 12666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:56.804193 12666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:56.807749 12666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "445c1f26cfeb487a9bd42c295bba802a"
format_stamp: "Formatted at 2026-09-28 16:49:56 on dist-test-slave-wzcb"
server_key: "41880c63c6f4eb8c4f5c386a459ed6e1"
server_key_iv: "231372aa6765c97b8d155d60999dcb96"
server_key_version: "encryptionkey@0"
I20260928 16:49:56.808656 12666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "445c1f26cfeb487a9bd42c295bba802a"
format_stamp: "Formatted at 2026-09-28 16:49:56 on dist-test-slave-wzcb"
server_key: "41880c63c6f4eb8c4f5c386a459ed6e1"
server_key_iv: "231372aa6765c97b8d155d60999dcb96"
server_key_version: "encryptionkey@0"
I20260928 16:49:56.814919 12666 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 16:49:56.819597 12681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:56.821642 12666 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:49:56.821903 12666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "445c1f26cfeb487a9bd42c295bba802a"
format_stamp: "Formatted at 2026-09-28 16:49:56 on dist-test-slave-wzcb"
server_key: "41880c63c6f4eb8c4f5c386a459ed6e1"
server_key_iv: "231372aa6765c97b8d155d60999dcb96"
server_key_version: "encryptionkey@0"
I20260928 16:49:56.822137 12666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:56.840778 12666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:56.845568 12666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:56.845872 12666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:56.864976 12666 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:37697
I20260928 16:49:56.864997 12732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:37697 every 8 connection(s)
I20260928 16:49:56.866526 12666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:49:56.870858 12733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:56.875845 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 12666
I20260928 16:49:56.876055 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:49:56.876740 9588 external_mini_cluster.cc:1428] Setting key 6ba22649ecdec1a6657612406fb4fccb
I20260928 16:49:56.885208 12733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 445c1f26cfeb487a9bd42c295bba802a: Bootstrap starting.
I20260928 16:49:56.889618 12733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 445c1f26cfeb487a9bd42c295bba802a: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:56.891211 12733 log.cc:824] T 00000000000000000000000000000000 P 445c1f26cfeb487a9bd42c295bba802a: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:49:56 dist-test-slave-wzcb krb5kdc[12653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:49:56.895274 12733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 445c1f26cfeb487a9bd42c295bba802a: No bootstrap required, opened a new log
I20260928 16:49:56.906134 12736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:56.879169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57610 (local address 127.9.93.62:37697)
0928 16:49:56.880161 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:56.880180 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:56.880250 (+ 70us) server_negotiation.cc:407] Connection header received
0928 16:49:56.882911 (+ 2661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:56.882953 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:56.883677 (+ 724us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:56.884217 (+ 540us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:56.885685 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:56.887816 (+ 2131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:56.889262 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:56.890434 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:56.890650 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:56.895673 (+ 5023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:56.895689 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:56.895715 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:56.895770 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:56.900380 (+ 4610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:56.901338 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:56.901353 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:56.901364 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:56.901663 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:56.902096 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:56.902101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:56.902104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:56.903016 (+ 912us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:56.903405 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:56.903817 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 16:49:56.904402 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":385,"threads_started":1}
I20260928 16:49:56.906776 12733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 445c1f26cfeb487a9bd42c295bba802a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "445c1f26cfeb487a9bd42c295bba802a" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 37697 } }
I20260928 16:49:56.907351 12733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 445c1f26cfeb487a9bd42c295bba802a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:56.907501 12733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 445c1f26cfeb487a9bd42c295bba802a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445c1f26cfeb487a9bd42c295bba802a, State: Initialized, Role: FOLLOWER
I20260928 16:49:56.908231 12733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 445c1f26cfeb487a9bd42c295bba802a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "445c1f26cfeb487a9bd42c295bba802a" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 37697 } }
I20260928 16:49:56.908495 12733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 445c1f26cfeb487a9bd42c295bba802a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:49:56.908650 12733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 445c1f26cfeb487a9bd42c295bba802a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:49:56.908842 12733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 445c1f26cfeb487a9bd42c295bba802a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:56.911607 12733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 445c1f26cfeb487a9bd42c295bba802a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "445c1f26cfeb487a9bd42c295bba802a" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 37697 } }
I20260928 16:49:56.912215 12733 leader_election.cc:302] T 00000000000000000000000000000000 P 445c1f26cfeb487a9bd42c295bba802a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 445c1f26cfeb487a9bd42c295bba802a; no voters:
I20260928 16:49:56.912837 12733 leader_election.cc:288] T 00000000000000000000000000000000 P 445c1f26cfeb487a9bd42c295bba802a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:49:56.912989 12738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 445c1f26cfeb487a9bd42c295bba802a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:56.913689 12738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 445c1f26cfeb487a9bd42c295bba802a [term 1 LEADER]: Becoming Leader. State: Replica: 445c1f26cfeb487a9bd42c295bba802a, State: Running, Role: LEADER
I20260928 16:49:56.914279 12738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 445c1f26cfeb487a9bd42c295bba802a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "445c1f26cfeb487a9bd42c295bba802a" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 37697 } }
I20260928 16:49:56.915136 12733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 445c1f26cfeb487a9bd42c295bba802a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:56.917212 12740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 445c1f26cfeb487a9bd42c295bba802a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 445c1f26cfeb487a9bd42c295bba802a. Latest consensus state: current_term: 1 leader_uuid: "445c1f26cfeb487a9bd42c295bba802a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "445c1f26cfeb487a9bd42c295bba802a" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 37697 } } }
I20260928 16:49:56.917716 12740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 445c1f26cfeb487a9bd42c295bba802a [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:56.917568 12739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 445c1f26cfeb487a9bd42c295bba802a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "445c1f26cfeb487a9bd42c295bba802a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "445c1f26cfeb487a9bd42c295bba802a" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 37697 } } }
I20260928 16:49:56.917883 12739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 445c1f26cfeb487a9bd42c295bba802a [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:56.920643 12744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:49:56.933944 12744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:49:56.945048 12744 catalog_manager.cc:1347] Generated new cluster ID: 8e44abdd4adf4afb8d6debe43ed5abac
I20260928 16:49:56.945204 12744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:49:56.960722 12744 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:49:56.962287 12744 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:49:56.974807 12744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 445c1f26cfeb487a9bd42c295bba802a: Generated new TSK 0
I20260928 16:49:56.975914 12744 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
I20260928 16:49:57.044550 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:37697
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:44531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:57.421025 12759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:57.421362 12759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:57.421517 12759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:57.429736 12759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:57.429888 12759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:57.430060 12759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:49:57.440970 12759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:44531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:37697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:57.442510 12759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:57.444090 12759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:57.458504 12764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:57.458544 12767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:57.458616 12765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:57.459547 12759 server_base.cc:1034] running on GCE node
I20260928 16:49:57.460290 12759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:57.461473 12759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:57.462697 12759 hybrid_clock.cc:648] HybridClock initialized: now 1790614197462633 us; error 55 us; skew 500 ppm
I20260928 16:49:57.463074 12759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:57 dist-test-slave-wzcb krb5kdc[12653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614197, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:57.466626 12759 init.cc:377] Logged in from keytab as kudu/127.9.93.1@KRBTEST.COM (short username kudu)
I20260928 16:49:57.468463 12759 webserver.cc:460] Webserver started at http://127.9.93.1:46473/ using document root <none> and password file <none>
I20260928 16:49:57.469322 12759 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:57.469465 12759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:57.469818 12759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:57.473693 12759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "8a8f125003fe42d1a4adfa7ec5d35f81"
format_stamp: "Formatted at 2026-09-28 16:49:57 on dist-test-slave-wzcb"
server_key: "3fa83dea9bc6189cbb669af4f41f3b94"
server_key_iv: "b412dd7a85135175e3809d4a3505712c"
server_key_version: "encryptionkey@0"
I20260928 16:49:57.475215 12759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "8a8f125003fe42d1a4adfa7ec5d35f81"
format_stamp: "Formatted at 2026-09-28 16:49:57 on dist-test-slave-wzcb"
server_key: "3fa83dea9bc6189cbb669af4f41f3b94"
server_key_iv: "b412dd7a85135175e3809d4a3505712c"
server_key_version: "encryptionkey@0"
I20260928 16:49:57.483917 12759 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260928 16:49:57.489921 12774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:57.491981 12759 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 16:49:57.492201 12759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "8a8f125003fe42d1a4adfa7ec5d35f81"
format_stamp: "Formatted at 2026-09-28 16:49:57 on dist-test-slave-wzcb"
server_key: "3fa83dea9bc6189cbb669af4f41f3b94"
server_key_iv: "b412dd7a85135175e3809d4a3505712c"
server_key_version: "encryptionkey@0"
I20260928 16:49:57.492437 12759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:57.520634 12759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:57.526248 12759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:57.526583 12759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:57.527863 12759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:49:57.529531 12759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:49:57.529623 12759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:57.529757 12759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:49:57.529815 12759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:57.563467 12759 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:43889
I20260928 16:49:57.563623 12886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:43889 every 8 connection(s)
I20260928 16:49:57.564945 12759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:49:57.570865 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 12759
I20260928 16:49:57.571102 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:49:57.571841 9588 external_mini_cluster.cc:1428] Setting key 158217c0b1ec32b6914cb0dede3511be
Sep 28 16:49:57 dist-test-slave-wzcb krb5kdc[12653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614197, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:49:57.589111 12889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:57.568201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:57185 (local address 127.9.93.62:37697)
0928 16:49:57.568815 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:57.568820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:57.569971 (+ 1151us) server_negotiation.cc:407] Connection header received
0928 16:49:57.572306 (+ 2335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:57.572313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:57.572411 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:57.572526 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:57.574491 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:57.575242 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:57.577096 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:57.577535 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:57.577661 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:57.582020 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:57.582026 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:57.582029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:57.582070 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:57.585015 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:57.586186 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:57.586193 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:57.586197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:57.586383 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:57.586924 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:57.586929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:57.586932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:57.587361 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:57.587530 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:57.588613 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0928 16:49:57.588834 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":371,"threads_started":1}
I20260928 16:49:57.591003 12890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:57.568826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:37697 (local address 127.9.93.1:57185)
0928 16:49:57.569757 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:57.569811 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:57.571926 (+ 2115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:57.572847 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:57.572858 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:57.573529 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:57.574275 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:57.574303 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:57.575458 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:57.575462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:57.576895 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:57.576912 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:57.577822 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:57.577831 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:57.578113 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:57.578854 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:57.578881 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:57.581699 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:57.585221 (+ 3522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:57.585231 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:57.585248 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:57.586006 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:57.586533 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:57.586539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:57.586543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:57.586789 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:57.587532 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:57.587540 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:57.588144 (+ 604us) client_negotiation.cc:769] Sending connection context
0928 16:49:57.588489 (+ 345us) client_negotiation.cc:240] Negotiation successful
0928 16:49:57.589226 (+ 737us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":379,"threads_started":1}
WARNING: no policy specified for kudu/127.9.93.2@KRBTEST.COM; defaulting to no policy
I20260928 16:49:57.592906 12887 heartbeater.cc:347] Connected to a master server at 127.9.93.62:37697
I20260928 16:49:57.593287 12887 heartbeater.cc:464] Registering TS with master...
I20260928 16:49:57.594246 12887 heartbeater.cc:511] Master 127.9.93.62:37697 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.2@KRBTEST.COM" created.
I20260928 16:49:57.597236 12698 ts_manager.cc:194] Registered new tserver with Master: 8a8f125003fe42d1a4adfa7ec5d35f81 (127.9.93.1:43889)
I20260928 16:49:57.599568 12698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.1@KRBTEST.COM'} at 127.9.93.1:57185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
I20260928 16:49:57.623147 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:37697
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:44531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:58.006539 12893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:58.006930 12893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:58.007088 12893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:58.015431 12893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:58.015607 12893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:58.015838 12893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:49:58.027242 12893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:44531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:37697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:58.028895 12893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:58.030573 12893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:58.046088 12898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:58.046156 12899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:58.046097 12901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:58.047173 12893 server_base.cc:1034] running on GCE node
I20260928 16:49:58.047977 12893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:58.049182 12893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:58.050423 12893 hybrid_clock.cc:648] HybridClock initialized: now 1790614198050353 us; error 53 us; skew 500 ppm
I20260928 16:49:58.050836 12893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:58 dist-test-slave-wzcb krb5kdc[12653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614198, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:58.060474 12893 init.cc:377] Logged in from keytab as kudu/127.9.93.2@KRBTEST.COM (short username kudu)
I20260928 16:49:58.062458 12893 webserver.cc:460] Webserver started at http://127.9.93.2:43525/ using document root <none> and password file <none>
I20260928 16:49:58.063444 12893 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:58.063593 12893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:58.063957 12893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:58.067780 12893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "1347944c0b7248d19d8f5c84e882cc82"
format_stamp: "Formatted at 2026-09-28 16:49:58 on dist-test-slave-wzcb"
server_key: "02180bb7737b2c25d7539b3540bbcfc9"
server_key_iv: "7a5d635069cf40ee75c52dd30db8a9a9"
server_key_version: "encryptionkey@0"
I20260928 16:49:58.068809 12893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "1347944c0b7248d19d8f5c84e882cc82"
format_stamp: "Formatted at 2026-09-28 16:49:58 on dist-test-slave-wzcb"
server_key: "02180bb7737b2c25d7539b3540bbcfc9"
server_key_iv: "7a5d635069cf40ee75c52dd30db8a9a9"
server_key_version: "encryptionkey@0"
I20260928 16:49:58.075546 12893 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 16:49:58.080102 12908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:58.081949 12893 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:49:58.082196 12893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "1347944c0b7248d19d8f5c84e882cc82"
format_stamp: "Formatted at 2026-09-28 16:49:58 on dist-test-slave-wzcb"
server_key: "02180bb7737b2c25d7539b3540bbcfc9"
server_key_iv: "7a5d635069cf40ee75c52dd30db8a9a9"
server_key_version: "encryptionkey@0"
I20260928 16:49:58.082456 12893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:58.096684 12893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:58.102335 12893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:58.102736 12893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:58.104143 12893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:49:58.106029 12893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:49:58.106132 12893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:58.106276 12893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:49:58.106336 12893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:58.142475 12893 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:40805
I20260928 16:49:58.142515 13020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:40805 every 8 connection(s)
I20260928 16:49:58.144053 12893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:49:58.147022 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 12893
I20260928 16:49:58.147235 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:49:58.147840 9588 external_mini_cluster.cc:1428] Setting key 2832219d5951060ffd79b11f6a91e5e3
Sep 28 16:49:58 dist-test-slave-wzcb krb5kdc[12653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614198, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.93.3@KRBTEST.COM; defaulting to no policy
I20260928 16:49:58.171840 13023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:58.147235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:45339 (local address 127.9.93.62:37697)
0928 16:49:58.147818 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:58.147826 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:58.149130 (+ 1304us) server_negotiation.cc:407] Connection header received
0928 16:49:58.151779 (+ 2649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:58.151784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:58.151858 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:58.151960 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:58.154502 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:58.155334 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:58.157949 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:58.158674 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:58.158837 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:58.163498 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:58.163506 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:58.163511 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:58.163562 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:58.167296 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:58.168745 (+ 1449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:58.168751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:58.168753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:58.168922 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:58.169552 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:58.169556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:58.169558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:58.169855 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:58.169989 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:58.171183 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0928 16:49:58.171510 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":263,"threads_started":1}
I20260928 16:49:58.173095 13024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:58.147915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:37697 (local address 127.9.93.2:45339)
0928 16:49:58.148933 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:58.149008 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:58.151395 (+ 2387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:58.152321 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:58.152338 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:58.153232 (+ 894us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:58.154252 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:58.154290 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:58.155573 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:58.155579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:58.157687 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:58.157710 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:58.159012 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:58.159019 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:58.159283 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:58.160134 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:58.160165 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:58.163102 (+ 2937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:58.167590 (+ 4488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:58.167605 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:58.167632 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:58.168549 (+ 917us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:58.169068 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:58.169074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:58.169078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:58.169423 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:58.170012 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:58.170022 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:58.170630 (+ 608us) client_negotiation.cc:769] Sending connection context
0928 16:49:58.171048 (+ 418us) client_negotiation.cc:240] Negotiation successful
0928 16:49:58.171783 (+ 735us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":361,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.3@KRBTEST.COM" created.
I20260928 16:49:58.175088 13021 heartbeater.cc:347] Connected to a master server at 127.9.93.62:37697
I20260928 16:49:58.175626 13021 heartbeater.cc:464] Registering TS with master...
I20260928 16:49:58.176864 13021 heartbeater.cc:511] Master 127.9.93.62:37697 requested a full tablet report, sending...
I20260928 16:49:58.179193 12698 ts_manager.cc:194] Registered new tserver with Master: 1347944c0b7248d19d8f5c84e882cc82 (127.9.93.2:40805)
I20260928 16:49:58.180202 12698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.2@KRBTEST.COM'} at 127.9.93.2:45339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
I20260928 16:49:58.203186 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:37697
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:44531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:58.574802 13027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:58.575160 13027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:58.575327 13027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:58.583603 13027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:58.583750 13027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:58.583930 13027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:49:58.595191 13027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:44531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:37697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:58.596829 13027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:58.598416 13027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:49:58.603824 12887 heartbeater.cc:503] Master 127.9.93.62:37697 was elected leader, sending a full tablet report...
W20260928 16:49:58.613821 13032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:58.613821 13035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:58.613871 13033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:58.614792 13027 server_base.cc:1034] running on GCE node
I20260928 16:49:58.615638 13027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:58.616814 13027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:58.618058 13027 hybrid_clock.cc:648] HybridClock initialized: now 1790614198617990 us; error 51 us; skew 500 ppm
I20260928 16:49:58.618450 13027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:58 dist-test-slave-wzcb krb5kdc[12653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614198, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:58.622757 13027 init.cc:377] Logged in from keytab as kudu/127.9.93.3@KRBTEST.COM (short username kudu)
I20260928 16:49:58.624913 13027 webserver.cc:460] Webserver started at http://127.9.93.3:46081/ using document root <none> and password file <none>
I20260928 16:49:58.625806 13027 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:58.625952 13027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:58.626329 13027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:58.629977 13027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "aad9a1bd259142fbb3b7f765aec5b781"
format_stamp: "Formatted at 2026-09-28 16:49:58 on dist-test-slave-wzcb"
server_key: "5798291e5e7c2bc28751609314307e1d"
server_key_iv: "a8fa8c3281c1678767f6bbb49f110015"
server_key_version: "encryptionkey@0"
I20260928 16:49:58.630927 13027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "aad9a1bd259142fbb3b7f765aec5b781"
format_stamp: "Formatted at 2026-09-28 16:49:58 on dist-test-slave-wzcb"
server_key: "5798291e5e7c2bc28751609314307e1d"
server_key_iv: "a8fa8c3281c1678767f6bbb49f110015"
server_key_version: "encryptionkey@0"
I20260928 16:49:58.637244 13027 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 16:49:58.641676 13042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:58.643532 13027 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:49:58.643750 13027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "aad9a1bd259142fbb3b7f765aec5b781"
format_stamp: "Formatted at 2026-09-28 16:49:58 on dist-test-slave-wzcb"
server_key: "5798291e5e7c2bc28751609314307e1d"
server_key_iv: "a8fa8c3281c1678767f6bbb49f110015"
server_key_version: "encryptionkey@0"
I20260928 16:49:58.643981 13027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:58.679008 13027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:58.684240 13027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:58.684594 13027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:58.685756 13027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:49:58.687513 13027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:49:58.687608 13027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:58.687745 13027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:49:58.687803 13027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:58.722792 13027 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:44939
I20260928 16:49:58.722862 13154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:44939 every 8 connection(s)
I20260928 16:49:58.724363 13027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:49:58.726832 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 13027
I20260928 16:49:58.727037 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:49:58.727605 9588 external_mini_cluster.cc:1428] Setting key 7db20334745601e8ad7b4ab93e1a5437
Sep 28 16:49:58 dist-test-slave-wzcb krb5kdc[12653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614198, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:49:58.748953 13157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:58.727912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:44741 (local address 127.9.93.62:37697)
0928 16:49:58.731126 (+ 3214us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:58.731132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:58.731154 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:49:58.732698 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:58.732702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:58.732793 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:58.732896 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:58.734780 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:58.735763 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:58.737597 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:58.738056 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:58.738174 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:58.742482 (+ 4308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:58.742489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:58.742495 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:58.742539 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:58.745456 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:58.746323 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:58.746327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:58.746329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:58.746458 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:58.746905 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:58.746908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:58.746910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:58.747216 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:58.747372 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:58.748376 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0928 16:49:58.748609 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3101,"thread_start_us":2942,"threads_started":1}
I20260928 16:49:58.750300 13158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:58.728924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:37697 (local address 127.9.93.3:44741)
0928 16:49:58.730150 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:58.730231 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:58.732371 (+ 2140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:58.733213 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:58.733223 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:58.733861 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:58.734559 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:58.734580 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:58.735966 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:58.735971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:58.737412 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:58.737429 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:58.738342 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:58.738348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:58.738639 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:58.739410 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:58.739448 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:58.742213 (+ 2765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:58.745630 (+ 3417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:58.745639 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:58.745658 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:58.746213 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:58.746563 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:58.746567 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:58.746569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:58.746816 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:58.747351 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:58.747360 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:58.747930 (+ 570us) client_negotiation.cc:769] Sending connection context
0928 16:49:58.748257 (+ 327us) client_negotiation.cc:240] Negotiation successful
0928 16:49:58.748894 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":485,"threads_started":1}
I20260928 16:49:58.752049 13155 heartbeater.cc:347] Connected to a master server at 127.9.93.62:37697
I20260928 16:49:58.752425 13155 heartbeater.cc:464] Registering TS with master...
I20260928 16:49:58.753299 13155 heartbeater.cc:511] Master 127.9.93.62:37697 requested a full tablet report, sending...
I20260928 16:49:58.755007 12698 ts_manager.cc:194] Registered new tserver with Master: aad9a1bd259142fbb3b7f765aec5b781 (127.9.93.3:44939)
I20260928 16:49:58.756316 12698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.3@KRBTEST.COM'} at 127.9.93.3:44741
I20260928 16:49:58.757174 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 16:49:58 dist-test-slave-wzcb krb5kdc[12653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614198, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 16:49:58 dist-test-slave-wzcb krb5kdc[12653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614198, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:49:58.810384 13157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:58.795220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57618 (local address 127.9.93.62:37697)
0928 16:49:58.795414 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:58.795420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:58.795943 (+ 523us) server_negotiation.cc:407] Connection header received
0928 16:49:58.796129 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:58.796134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:58.796221 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:58.796314 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:58.797874 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:58.798693 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:58.800093 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:58.800531 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:58.800651 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:58.804833 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:58.804838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:58.804842 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:58.804885 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:58.807615 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:58.808310 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:58.808315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:58.808317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:58.808447 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:58.808906 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:58.808911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:58.808913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:58.809327 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:58.809543 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:58.809915 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 16:49:58.810133 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 16:49:58.810998 12698 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57618
I20260928 16:49:58.812871 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 12759
Sep 28 16:49:58 dist-test-slave-wzcb krb5kdc[12653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614198, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.93.1@KRBTEST.COM
I20260928 16:49:58.830920 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:58.814073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54190 (local address 127.9.93.1:43889)
0928 16:49:58.814760 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:58.814768 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:58.814816 (+ 48us) server_negotiation.cc:407] Connection header received
0928 16:49:58.814885 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:58.814892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:58.815166 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:58.815422 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:58.816797 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:58.817899 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:58.819288 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:58.819986 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:58.820179 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:58.824430 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:58.824447 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:58.824471 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:58.824517 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:58.827736 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:58.828562 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:58.828570 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:58.828573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:58.828735 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:58.829111 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:58.829116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:58.829118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:58.829620 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:58.829930 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:58.830189 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 16:49:58.830446 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":345,"threads_started":1}
W20260928 16:49:58.832588 12882 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54190
W20260928 16:49:58.833913 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 12759: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:49:58.834175 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 12759
I20260928 16:49:58.848775 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 12893
Sep 28 16:49:58 dist-test-slave-wzcb krb5kdc[12653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614198, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.93.2@KRBTEST.COM
I20260928 16:49:58.866047 13168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:58.849892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45666 (local address 127.9.93.2:40805)
0928 16:49:58.850558 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:58.850568 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:58.850599 (+ 31us) server_negotiation.cc:407] Connection header received
0928 16:49:58.850673 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:58.850680 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:58.850926 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:58.851071 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:58.852595 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:58.853820 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:58.855159 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:58.855914 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:58.856041 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:58.860226 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:58.860243 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:58.860261 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:58.860309 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:58.863081 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:58.863873 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:58.863883 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:58.863885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:58.864024 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:58.864343 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:58.864346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:58.864348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:58.864758 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:58.865015 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:58.865413 (+ 398us) server_negotiation.cc:299] Negotiation successful
0928 16:49:58.865680 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":319,"threads_started":1}
W20260928 16:49:58.867697 13016 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45666
W20260928 16:49:58.868949 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 12893: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:49:58.869097 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 12893
I20260928 16:49:58.883018 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 13027
Sep 28 16:49:58 dist-test-slave-wzcb krb5kdc[12653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614198, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.93.3@KRBTEST.COM
I20260928 16:49:58.900949 13169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:58.884075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51884 (local address 127.9.93.3:44939)
0928 16:49:58.884766 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:58.884774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:58.884808 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:49:58.884879 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:58.884885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:58.885123 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:58.885268 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:58.886623 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:58.887861 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:58.889155 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:58.889763 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:58.889864 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:58.894352 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:58.894371 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:58.894390 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:58.894442 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:58.897602 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:58.898408 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:58.898418 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:58.898421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:58.898636 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:58.899020 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:58.899025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:58.899027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:58.899662 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:58.899944 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:58.900279 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 16:49:58.900529 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":303,"threads_started":1}
W20260928 16:49:58.902480 13150 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51884
W20260928 16:49:58.903879 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 13027: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:49:58.904063 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 13027
I20260928 16:49:58.917486 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 12666
I20260928 16:49:58.918841 12728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:49:59.024557 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 12666
2026-09-28T16:49:59Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3857 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:49:59 dist-test-slave-wzcb krb5kdc[13173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:49:59 dist-test-slave-wzcb krb5kdc[13173](info): set up 2 sockets
Sep 28 16:49:59 dist-test-slave-wzcb krb5kdc[13173](info): commencing operation
krb5kdc: starting...
W20260928 16:50:00.038240 9588 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:50:00 dist-test-slave-wzcb krb5kdc[13173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:50:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:50:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:50:00.257121 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:44163
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:33299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:44163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:50:00.612305 13186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:00.612659 13186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:00.612777 13186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:00.620766 13186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:50:00.620904 13186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:00.620954 13186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:00.621037 13186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:50:00.621095 13186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:50:00.633101 13186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:33299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:44163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:44163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:00.634675 13186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:00.636296 13186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:00.649474 13191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:00.649538 13194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:00.649534 13192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:00.650985 13186 server_base.cc:1034] running on GCE node
I20260928 16:50:00.652060 13186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:00.653774 13186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:00.655022 13186 hybrid_clock.cc:648] HybridClock initialized: now 1790614200654949 us; error 68 us; skew 500 ppm
I20260928 16:50:00.655457 13186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:00 dist-test-slave-wzcb krb5kdc[13173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614200, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:00.658985 13186 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:50:00.660751 13186 webserver.cc:460] Webserver started at http://127.9.93.62:39015/ using document root <none> and password file <none>
I20260928 16:50:00.661586 13186 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:00.661708 13186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:00.662015 13186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:00.665465 13186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "38771717cb7041d39eda782aee4ce952"
format_stamp: "Formatted at 2026-09-28 16:50:00 on dist-test-slave-wzcb"
server_key: "7b474c3a95e68f69d45280bca3db07ce"
server_key_iv: "0d85e67fe7175c7748059b0a7bd94443"
server_key_version: "encryptionkey@0"
I20260928 16:50:00.666352 13186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "38771717cb7041d39eda782aee4ce952"
format_stamp: "Formatted at 2026-09-28 16:50:00 on dist-test-slave-wzcb"
server_key: "7b474c3a95e68f69d45280bca3db07ce"
server_key_iv: "0d85e67fe7175c7748059b0a7bd94443"
server_key_version: "encryptionkey@0"
I20260928 16:50:00.672639 13186 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 16:50:00.677089 13201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:00.678968 13186 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:50:00.679170 13186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "38771717cb7041d39eda782aee4ce952"
format_stamp: "Formatted at 2026-09-28 16:50:00 on dist-test-slave-wzcb"
server_key: "7b474c3a95e68f69d45280bca3db07ce"
server_key_iv: "0d85e67fe7175c7748059b0a7bd94443"
server_key_version: "encryptionkey@0"
I20260928 16:50:00.679446 13186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:00.707881 13186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:00.713083 13186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:00.713379 13186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:00.733208 13186 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:44163
I20260928 16:50:00.733242 13252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:44163 every 8 connection(s)
I20260928 16:50:00.734779 13186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:50:00.739238 13253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:00.740231 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 13186
I20260928 16:50:00.740512 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:50:00.741176 9588 external_mini_cluster.cc:1428] Setting key 516d6610bfcca543fe78aa9689f12de4
I20260928 16:50:00.755129 13253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38771717cb7041d39eda782aee4ce952: Bootstrap starting.
Sep 28 16:50:00 dist-test-slave-wzcb krb5kdc[13173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:50:00.759331 13253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38771717cb7041d39eda782aee4ce952: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:00.760659 13253 log.cc:824] T 00000000000000000000000000000000 P 38771717cb7041d39eda782aee4ce952: Log is configured to *not* fsync() on all Append() calls
I20260928 16:50:00.764101 13253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38771717cb7041d39eda782aee4ce952: No bootstrap required, opened a new log
I20260928 16:50:00.766862 13256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:00.743430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56252 (local address 127.9.93.62:44163)
0928 16:50:00.744373 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:00.744390 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:00.744445 (+ 55us) server_negotiation.cc:407] Connection header received
0928 16:50:00.746265 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:00.746303 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:00.746833 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:00.747291 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:00.749398 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:00.750845 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:00.752803 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:00.753573 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:00.753715 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:00.758791 (+ 5076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:00.758801 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:00.758819 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:00.758875 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:00.762157 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:00.763015 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:00.763023 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:00.763031 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:00.763232 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:00.763681 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:00.763685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:00.763687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:00.764290 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:00.764627 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:00.765107 (+ 480us) server_negotiation.cc:299] Negotiation successful
0928 16:50:00.765625 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":418,"threads_started":1}
I20260928 16:50:00.772668 13253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38771717cb7041d39eda782aee4ce952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38771717cb7041d39eda782aee4ce952" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 44163 } }
I20260928 16:50:00.773159 13253 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38771717cb7041d39eda782aee4ce952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:00.773273 13253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38771717cb7041d39eda782aee4ce952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38771717cb7041d39eda782aee4ce952, State: Initialized, Role: FOLLOWER
I20260928 16:50:00.773958 13253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38771717cb7041d39eda782aee4ce952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38771717cb7041d39eda782aee4ce952" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 44163 } }
I20260928 16:50:00.774233 13253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38771717cb7041d39eda782aee4ce952 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:50:00.774369 13253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38771717cb7041d39eda782aee4ce952 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:50:00.774546 13253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38771717cb7041d39eda782aee4ce952 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:00.777043 13253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38771717cb7041d39eda782aee4ce952 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38771717cb7041d39eda782aee4ce952" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 44163 } }
I20260928 16:50:00.777668 13253 leader_election.cc:302] T 00000000000000000000000000000000 P 38771717cb7041d39eda782aee4ce952 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38771717cb7041d39eda782aee4ce952; no voters:
I20260928 16:50:00.778337 13253 leader_election.cc:288] T 00000000000000000000000000000000 P 38771717cb7041d39eda782aee4ce952 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:50:00.778498 13258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38771717cb7041d39eda782aee4ce952 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:50:00.779228 13258 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38771717cb7041d39eda782aee4ce952 [term 1 LEADER]: Becoming Leader. State: Replica: 38771717cb7041d39eda782aee4ce952, State: Running, Role: LEADER
I20260928 16:50:00.779901 13258 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38771717cb7041d39eda782aee4ce952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38771717cb7041d39eda782aee4ce952" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 44163 } }
I20260928 16:50:00.780840 13253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38771717cb7041d39eda782aee4ce952 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:50:00.783053 13260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38771717cb7041d39eda782aee4ce952 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38771717cb7041d39eda782aee4ce952. Latest consensus state: current_term: 1 leader_uuid: "38771717cb7041d39eda782aee4ce952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38771717cb7041d39eda782aee4ce952" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 44163 } } }
I20260928 16:50:00.783609 13260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38771717cb7041d39eda782aee4ce952 [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:00.783185 13259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38771717cb7041d39eda782aee4ce952 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38771717cb7041d39eda782aee4ce952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38771717cb7041d39eda782aee4ce952" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 44163 } } }
I20260928 16:50:00.785089 13259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38771717cb7041d39eda782aee4ce952 [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:00.786322 13266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:50:00.792299 13266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:50:00.803686 13266 catalog_manager.cc:1347] Generated new cluster ID: ec6f222de3454a90866493bcb4b1b680
I20260928 16:50:00.803814 13266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:50:00.813741 13266 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:50:00.815186 13266 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:50:00.829187 13266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38771717cb7041d39eda782aee4ce952: Generated new TSK 0
I20260928 16:50:00.830572 13266 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
I20260928 16:50:00.893721 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:44163
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:33299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:50:01.282191 13279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:01.282546 13279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:01.282703 13279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:01.291821 13279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:01.291978 13279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:01.292155 13279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:50:01.303543 13279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:33299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:44163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:01.305099 13279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:01.306730 13279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:01.321732 13285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:01.321774 13284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:01.321693 13287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:01.323138 13279 server_base.cc:1034] running on GCE node
I20260928 16:50:01.324054 13279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:01.325291 13279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:01.326557 13279 hybrid_clock.cc:648] HybridClock initialized: now 1790614201326481 us; error 61 us; skew 500 ppm
I20260928 16:50:01.326983 13279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:01 dist-test-slave-wzcb krb5kdc[13173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614201, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:01.330941 13279 init.cc:377] Logged in from keytab as kudu/127.9.93.1@KRBTEST.COM (short username kudu)
I20260928 16:50:01.332614 13279 webserver.cc:460] Webserver started at http://127.9.93.1:37705/ using document root <none> and password file <none>
I20260928 16:50:01.333400 13279 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:01.333523 13279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:01.333829 13279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:01.337277 13279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "092231d83d41464ca41ba04c82648d2d"
format_stamp: "Formatted at 2026-09-28 16:50:01 on dist-test-slave-wzcb"
server_key: "7bf41111892aebde570b643013de5072"
server_key_iv: "a9c78e6d60948564df33a06b391a8ef1"
server_key_version: "encryptionkey@0"
I20260928 16:50:01.338308 13279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "092231d83d41464ca41ba04c82648d2d"
format_stamp: "Formatted at 2026-09-28 16:50:01 on dist-test-slave-wzcb"
server_key: "7bf41111892aebde570b643013de5072"
server_key_iv: "a9c78e6d60948564df33a06b391a8ef1"
server_key_version: "encryptionkey@0"
I20260928 16:50:01.344525 13279 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 16:50:01.348835 13294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:01.350654 13279 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:50:01.350862 13279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "092231d83d41464ca41ba04c82648d2d"
format_stamp: "Formatted at 2026-09-28 16:50:01 on dist-test-slave-wzcb"
server_key: "7bf41111892aebde570b643013de5072"
server_key_iv: "a9c78e6d60948564df33a06b391a8ef1"
server_key_version: "encryptionkey@0"
I20260928 16:50:01.351100 13279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:01.378147 13279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:01.383096 13279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:01.383460 13279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:01.384673 13279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:50:01.386228 13279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:01.386308 13279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:01.386440 13279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:01.386493 13279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:01.420948 13279 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:34193
I20260928 16:50:01.421000 13406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:34193 every 8 connection(s)
I20260928 16:50:01.422456 13279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:50:01.429275 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 13279
I20260928 16:50:01.429507 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:50:01.430197 9588 external_mini_cluster.cc:1428] Setting key 51de3b3ba300c1f47d214e1a39f47a58
Sep 28 16:50:01 dist-test-slave-wzcb krb5kdc[13173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614201, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:50:01.445360 13409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:01.425387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:59227 (local address 127.9.93.62:44163)
0928 16:50:01.425853 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:01.425858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:01.427098 (+ 1240us) server_negotiation.cc:407] Connection header received
0928 16:50:01.429341 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:01.429347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:01.429465 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:01.429577 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:01.431635 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:01.432606 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:01.434359 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:01.434816 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:01.434938 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:01.439048 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:01.439054 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:01.439057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:01.439097 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:01.441827 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:01.442714 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:01.442719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:01.442722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:01.442857 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:01.443366 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:01.443370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:01.443373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:01.443675 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:01.443821 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:01.444676 (+ 855us) server_negotiation.cc:299] Negotiation successful
0928 16:50:01.444949 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":245,"threads_started":1}
I20260928 16:50:01.446520 13410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:01.425978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:44163 (local address 127.9.93.1:59227)
0928 16:50:01.426884 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:01.426942 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:01.428970 (+ 2028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:01.429897 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:01.429916 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:01.430575 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:01.431444 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:01.431469 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:01.432774 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:01.432779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:01.434181 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:01.434197 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:01.435098 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:01.435107 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:01.435412 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:01.436210 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:01.436238 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:01.438759 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:01.442007 (+ 3248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:01.442018 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:01.442039 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:01.442561 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:01.442972 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:01.442977 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:01.442982 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:01.443212 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:01.443821 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:01.443830 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:01.444248 (+ 418us) client_negotiation.cc:769] Sending connection context
0928 16:50:01.444518 (+ 270us) client_negotiation.cc:240] Negotiation successful
0928 16:50:01.445193 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":364,"threads_started":1}
I20260928 16:50:01.448639 13407 heartbeater.cc:347] Connected to a master server at 127.9.93.62:44163
I20260928 16:50:01.448987 13407 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:01.449841 13407 heartbeater.cc:511] Master 127.9.93.62:44163 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.93.2@KRBTEST.COM; defaulting to no policy
I20260928 16:50:01.452491 13218 ts_manager.cc:194] Registered new tserver with Master: 092231d83d41464ca41ba04c82648d2d (127.9.93.1:34193)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.2@KRBTEST.COM" created.
I20260928 16:50:01.454167 13218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.1@KRBTEST.COM'} at 127.9.93.1:59227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
I20260928 16:50:01.479418 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:44163
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:33299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:50:01.855502 13413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:01.855979 13413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:01.856156 13413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:01.864492 13413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:01.864651 13413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:01.864827 13413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:50:01.876052 13413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:33299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:44163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:01.877629 13413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:01.879252 13413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:01.894249 13419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:01.894232 13418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:01.895083 13421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:01.895226 13413 server_base.cc:1034] running on GCE node
I20260928 16:50:01.896037 13413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:01.897176 13413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:01.898424 13413 hybrid_clock.cc:648] HybridClock initialized: now 1790614201898344 us; error 60 us; skew 500 ppm
I20260928 16:50:01.898850 13413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:01 dist-test-slave-wzcb krb5kdc[13173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614201, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:01.902421 13413 init.cc:377] Logged in from keytab as kudu/127.9.93.2@KRBTEST.COM (short username kudu)
I20260928 16:50:01.904232 13413 webserver.cc:460] Webserver started at http://127.9.93.2:44655/ using document root <none> and password file <none>
I20260928 16:50:01.905071 13413 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:01.905206 13413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:01.905555 13413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:01.909162 13413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "8f074ce1b6fa4da2a0022db870e58822"
format_stamp: "Formatted at 2026-09-28 16:50:01 on dist-test-slave-wzcb"
server_key: "1f90d52bd84e3da1d4448a349a7d2e9d"
server_key_iv: "2dddf4d92af982fd17b2d2e2752a0bc5"
server_key_version: "encryptionkey@0"
I20260928 16:50:01.910092 13413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "8f074ce1b6fa4da2a0022db870e58822"
format_stamp: "Formatted at 2026-09-28 16:50:01 on dist-test-slave-wzcb"
server_key: "1f90d52bd84e3da1d4448a349a7d2e9d"
server_key_iv: "2dddf4d92af982fd17b2d2e2752a0bc5"
server_key_version: "encryptionkey@0"
I20260928 16:50:01.916487 13413 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 16:50:01.920938 13429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:01.922839 13413 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:50:01.923033 13413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "8f074ce1b6fa4da2a0022db870e58822"
format_stamp: "Formatted at 2026-09-28 16:50:01 on dist-test-slave-wzcb"
server_key: "1f90d52bd84e3da1d4448a349a7d2e9d"
server_key_iv: "2dddf4d92af982fd17b2d2e2752a0bc5"
server_key_version: "encryptionkey@0"
I20260928 16:50:01.923274 13413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:01.942637 13413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:01.947822 13413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:01.948148 13413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:01.949306 13413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:50:01.950971 13413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:01.951062 13413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:01.951197 13413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:01.951253 13413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:01.986479 13413 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:41949
I20260928 16:50:01.986523 13542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:41949 every 8 connection(s)
I20260928 16:50:01.988013 13413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:50:01.993309 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 13413
I20260928 16:50:01.993515 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:50:01.993947 9588 external_mini_cluster.cc:1428] Setting key 35baff01f264178bfe6ea01eb05704b7
Sep 28 16:50:01 dist-test-slave-wzcb krb5kdc[13173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614201, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:50:02.013044 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:01.991097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:47767 (local address 127.9.93.62:44163)
0928 16:50:01.991651 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:01.991657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:01.992939 (+ 1282us) server_negotiation.cc:407] Connection header received
0928 16:50:01.995197 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:01.995204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:01.995295 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:01.995427 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:01.997389 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:01.998177 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:02.000113 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:02.000605 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:02.000733 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:02.005197 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:02.005205 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:02.005210 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:02.005261 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:02.008885 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:02.009988 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:02.009995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:02.009998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:02.010201 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:02.010843 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:02.010848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:02.010851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:02.011225 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:02.011424 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:02.012308 (+ 884us) server_negotiation.cc:299] Negotiation successful
0928 16:50:02.012616 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":274,"threads_started":1}
I20260928 16:50:02.014106 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:01.991885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:44163 (local address 127.9.93.2:47767)
0928 16:50:01.992744 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:01.992799 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:01.994856 (+ 2057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:01.995777 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:01.995792 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:01.996460 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:01.997184 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:01.997206 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:01.998368 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:01.998373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:01.999912 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:01.999931 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:02.000908 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:02.000917 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:02.001193 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:02.001977 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:02.002013 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:02.004833 (+ 2820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:02.009132 (+ 4299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:02.009143 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:02.009161 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:02.009791 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:02.010383 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:02.010388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:02.010390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:02.010692 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:02.011388 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:02.011396 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:02.011885 (+ 489us) client_negotiation.cc:769] Sending connection context
0928 16:50:02.012146 (+ 261us) client_negotiation.cc:240] Negotiation successful
0928 16:50:02.012663 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":334,"threads_started":1}
WARNING: no policy specified for kudu/127.9.93.3@KRBTEST.COM; defaulting to no policy
I20260928 16:50:02.015951 13543 heartbeater.cc:347] Connected to a master server at 127.9.93.62:44163
I20260928 16:50:02.016295 13543 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:02.017246 13543 heartbeater.cc:511] Master 127.9.93.62:44163 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.3@KRBTEST.COM" created.
I20260928 16:50:02.019085 13218 ts_manager.cc:194] Registered new tserver with Master: 8f074ce1b6fa4da2a0022db870e58822 (127.9.93.2:41949)
I20260928 16:50:02.020074 13218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.2@KRBTEST.COM'} at 127.9.93.2:47767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
I20260928 16:50:02.048007 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:44163
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:33299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:50:02.416853 13549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:02.417196 13549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:02.417336 13549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:02.425747 13549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:02.425877 13549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:02.426052 13549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:50:02.437103 13549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:33299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:44163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:02.438656 13549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:02.440233 13549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:02.454869 13555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:02.454963 13554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:02.455001 13557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:02.455718 13549 server_base.cc:1034] running on GCE node
I20260928 16:50:02.456477 13549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:02.457666 13549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:02.458413 13407 heartbeater.cc:503] Master 127.9.93.62:44163 was elected leader, sending a full tablet report...
I20260928 16:50:02.458897 13549 hybrid_clock.cc:648] HybridClock initialized: now 1790614202458823 us; error 63 us; skew 500 ppm
I20260928 16:50:02.459296 13549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:02 dist-test-slave-wzcb krb5kdc[13173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614202, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:02.462939 13549 init.cc:377] Logged in from keytab as kudu/127.9.93.3@KRBTEST.COM (short username kudu)
I20260928 16:50:02.464828 13549 webserver.cc:460] Webserver started at http://127.9.93.3:43949/ using document root <none> and password file <none>
I20260928 16:50:02.465637 13549 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:02.465768 13549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:02.466101 13549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:02.469573 13549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "cbf96a859f0e4e099b9b1dd3e0475776"
format_stamp: "Formatted at 2026-09-28 16:50:02 on dist-test-slave-wzcb"
server_key: "cde9d6cc46d512886bc83fd0991e05b3"
server_key_iv: "8120ab593f81cfeed2f7403b53fa0562"
server_key_version: "encryptionkey@0"
I20260928 16:50:02.470475 13549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "cbf96a859f0e4e099b9b1dd3e0475776"
format_stamp: "Formatted at 2026-09-28 16:50:02 on dist-test-slave-wzcb"
server_key: "cde9d6cc46d512886bc83fd0991e05b3"
server_key_iv: "8120ab593f81cfeed2f7403b53fa0562"
server_key_version: "encryptionkey@0"
I20260928 16:50:02.476855 13549 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 16:50:02.481190 13564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:02.482805 13549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:50:02.482999 13549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "cbf96a859f0e4e099b9b1dd3e0475776"
format_stamp: "Formatted at 2026-09-28 16:50:02 on dist-test-slave-wzcb"
server_key: "cde9d6cc46d512886bc83fd0991e05b3"
server_key_iv: "8120ab593f81cfeed2f7403b53fa0562"
server_key_version: "encryptionkey@0"
I20260928 16:50:02.483225 13549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:02.503757 13549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:02.508807 13549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:02.509138 13549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:02.510281 13549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:50:02.511989 13549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:02.512079 13549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:02.512215 13549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:02.512277 13549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:02.546372 13549 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:44339
I20260928 16:50:02.546411 13677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:44339 every 8 connection(s)
I20260928 16:50:02.547937 13549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:50:02.551445 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 13549
I20260928 16:50:02.551677 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:50:02.552251 9588 external_mini_cluster.cc:1428] Setting key e7c3fce66cff38a241e215fab3342f99
Sep 28 16:50:02 dist-test-slave-wzcb krb5kdc[13173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614202, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:50:02.571810 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:02.550987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:43591 (local address 127.9.93.62:44163)
0928 16:50:02.551507 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:02.551516 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:02.553080 (+ 1564us) server_negotiation.cc:407] Connection header received
0928 16:50:02.555541 (+ 2461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:02.555547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:02.555643 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:02.555746 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:02.557630 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:02.558461 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:02.560275 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:02.560712 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:02.560831 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:02.564931 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:02.564937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:02.564941 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:02.564984 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:02.568193 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:02.569100 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:02.569105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:02.569107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:02.569245 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:02.569735 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:02.569738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:02.569741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:02.570033 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:02.570178 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:02.571063 (+ 885us) server_negotiation.cc:299] Negotiation successful
0928 16:50:02.571384 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":260,"threads_started":1}
I20260928 16:50:02.572993 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:02.551858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:44163 (local address 127.9.93.3:43591)
0928 16:50:02.552837 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:02.552902 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:02.555194 (+ 2292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:02.556048 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:02.556058 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:02.556744 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:02.557427 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:02.557448 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:02.558638 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:02.558642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:02.560110 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:02.560126 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:02.560989 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:02.560997 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:02.561261 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:02.561979 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:02.562012 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:02.564548 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:02.568380 (+ 3832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:02.568389 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:02.568406 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:02.568941 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:02.569370 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:02.569375 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:02.569377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:02.569608 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:02.570185 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:02.570194 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:02.570637 (+ 443us) client_negotiation.cc:769] Sending connection context
0928 16:50:02.570922 (+ 285us) client_negotiation.cc:240] Negotiation successful
0928 16:50:02.571679 (+ 757us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":425,"threads_started":1}
I20260928 16:50:02.574582 13678 heartbeater.cc:347] Connected to a master server at 127.9.93.62:44163
I20260928 16:50:02.574959 13678 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:02.575904 13678 heartbeater.cc:511] Master 127.9.93.62:44163 requested a full tablet report, sending...
I20260928 16:50:02.577634 13218 ts_manager.cc:194] Registered new tserver with Master: cbf96a859f0e4e099b9b1dd3e0475776 (127.9.93.3:44339)
I20260928 16:50:02.578542 13218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.3@KRBTEST.COM'} at 127.9.93.3:43591
I20260928 16:50:02.581207 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 16:50:02 dist-test-slave-wzcb krb5kdc[13173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 16:50:02 dist-test-slave-wzcb krb5kdc[13173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.1@KRBTEST.COM
I20260928 16:50:02.636458 13686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:02.619395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45388 (local address 127.9.93.1:34193)
0928 16:50:02.620126 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:02.620134 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:02.620162 (+ 28us) server_negotiation.cc:407] Connection header received
0928 16:50:02.620229 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:02.620236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:02.620466 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:02.620674 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:02.622235 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:02.623462 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:02.624768 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:02.625402 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:02.625530 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:02.630009 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:02.630026 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:02.630044 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:02.630094 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:02.633144 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:02.633874 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:02.633882 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:02.633884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:02.634032 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:02.634428 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:02.634431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:02.634433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:02.634950 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:02.635286 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:02.635716 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 16:50:02.636050 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":362,"threads_started":1}
Sep 28 16:50:02 dist-test-slave-wzcb krb5kdc[13173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:50:02.656563 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:02.641892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36186 (local address 127.9.93.62:44163)
0928 16:50:02.642125 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:02.642131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:02.642226 (+ 95us) server_negotiation.cc:407] Connection header received
0928 16:50:02.642386 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:02.642390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:02.642464 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:02.642538 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:02.643902 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:02.644676 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:02.645930 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:02.646368 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:02.646467 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:02.650835 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:02.650841 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:02.650845 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:02.650893 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:02.653501 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:02.654268 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:02.654273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:02.654276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:02.654437 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:02.654981 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:02.654984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:02.654987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:02.655434 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:02.655660 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:02.655994 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 16:50:02.656232 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260928 16:50:02.657236 13218 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36186
I20260928 16:50:02.658387 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 13279
I20260928 16:50:02.672231 13686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:02.659502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45394 (local address 127.9.93.1:34193)
0928 16:50:02.659735 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:02.659742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:02.659898 (+ 156us) server_negotiation.cc:407] Connection header received
0928 16:50:02.660060 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:02.660064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:02.660175 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:02.660259 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:02.661542 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:02.662294 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:02.663675 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:02.664233 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:02.664354 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:02.666852 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:02.666858 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:02.666862 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:02.666904 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:02.669624 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:02.670387 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:02.670391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:02.670394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:02.670550 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:02.670888 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:02.670891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:02.670894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:02.671184 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:02.671449 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:02.671697 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 16:50:02.671907 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 16:50:02.672976 13402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:02.766402 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 13279
I20260928 16:50:02.781684 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 13413
Sep 28 16:50:02 dist-test-slave-wzcb krb5kdc[13173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.2@KRBTEST.COM
I20260928 16:50:02.799779 13691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:02.782726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58922 (local address 127.9.93.2:41949)
0928 16:50:02.783394 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:02.783402 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:02.783433 (+ 31us) server_negotiation.cc:407] Connection header received
0928 16:50:02.783527 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:02.783534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:02.783830 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:02.784008 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:02.785359 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:02.786424 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:02.788073 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:02.788737 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:02.788893 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:02.793571 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:02.793589 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:02.793608 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:02.793664 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:02.796626 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:02.797408 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:02.797417 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:02.797420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:02.797569 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:02.797969 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:02.797974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:02.797976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:02.798399 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:02.798729 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:02.799068 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 16:50:02.799405 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":309,"threads_started":1}
I20260928 16:50:02.801155 13538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:02.888783 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 13413
I20260928 16:50:02.903537 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 13549
Sep 28 16:50:02 dist-test-slave-wzcb krb5kdc[13173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.3@KRBTEST.COM
I20260928 16:50:02.921595 13693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:02.904742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60740 (local address 127.9.93.3:44339)
0928 16:50:02.905473 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:02.905481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:02.905513 (+ 32us) server_negotiation.cc:407] Connection header received
0928 16:50:02.905589 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:02.905596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:02.905846 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:02.906015 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:02.907538 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:02.908661 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:02.910006 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:02.910643 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:02.910770 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:02.915362 (+ 4592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:02.915385 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:02.915404 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:02.915460 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:02.918324 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:02.919136 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:02.919146 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:02.919148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:02.919345 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:02.919788 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:02.919794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:02.919797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:02.920219 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:02.920542 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:02.920902 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 16:50:02.921222 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":384,"threads_started":1}
I20260928 16:50:02.923077 13673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:03.011726 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 13549
I20260928 16:50:03.026027 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 13186
I20260928 16:50:03.027359 13248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:03.133850 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 13186
2026-09-28T16:50:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4109 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:50:03 dist-test-slave-wzcb krb5kdc[13698](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:50:03 dist-test-slave-wzcb krb5kdc[13698](info): set up 2 sockets
Sep 28 16:50:03 dist-test-slave-wzcb krb5kdc[13698](info): commencing operation
krb5kdc: starting...
W20260928 16:50:04.150466 9588 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:50:04 dist-test-slave-wzcb krb5kdc[13698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:50:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:50:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:50:04.375483 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:42961
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:42961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:50:04.731259 13711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:04.731698 13711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:04.731802 13711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:04.739732 13711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:50:04.739869 13711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:04.739916 13711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:04.739974 13711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:50:04.740024 13711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:50:04.751048 13711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:42961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:42961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:04.752640 13711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:04.754155 13711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:04.767136 13716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:04.767181 13717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:04.767097 13719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:04.768143 13711 server_base.cc:1034] running on GCE node
I20260928 16:50:04.769140 13711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:04.770933 13711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:04.772207 13711 hybrid_clock.cc:648] HybridClock initialized: now 1790614204772118 us; error 90 us; skew 500 ppm
I20260928 16:50:04.772691 13711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:04 dist-test-slave-wzcb krb5kdc[13698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614204, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:04.776386 13711 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:50:04.777834 13711 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:04.777971 13711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:04.778337 13711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:04.781944 13711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "5dee33443f0d4e4cbfaf6f75d654389d"
format_stamp: "Formatted at 2026-09-28 16:50:04 on dist-test-slave-wzcb"
server_key: "4e74a3795f7e09bff760303847331cfe"
server_key_iv: "fe91dbf39aa384a0160383b922600abb"
server_key_version: "encryptionkey@0"
I20260928 16:50:04.782915 13711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "5dee33443f0d4e4cbfaf6f75d654389d"
format_stamp: "Formatted at 2026-09-28 16:50:04 on dist-test-slave-wzcb"
server_key: "4e74a3795f7e09bff760303847331cfe"
server_key_iv: "fe91dbf39aa384a0160383b922600abb"
server_key_version: "encryptionkey@0"
I20260928 16:50:04.789391 13711 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260928 16:50:04.794134 13725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:04.796046 13711 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:50:04.796289 13711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "5dee33443f0d4e4cbfaf6f75d654389d"
format_stamp: "Formatted at 2026-09-28 16:50:04 on dist-test-slave-wzcb"
server_key: "4e74a3795f7e09bff760303847331cfe"
server_key_iv: "fe91dbf39aa384a0160383b922600abb"
server_key_version: "encryptionkey@0"
I20260928 16:50:04.796552 13711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:04.820668 13711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:04.825559 13711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:04.825877 13711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:04.846135 13711 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:42961
I20260928 16:50:04.846168 13776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:42961 every 8 connection(s)
I20260928 16:50:04.847213 13711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:50:04.848734 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 13711
I20260928 16:50:04.848978 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:50:04.849825 9588 external_mini_cluster.cc:1428] Setting key 645e895375542395dd4a1a126d1936d4
I20260928 16:50:04.852874 13777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 16:50:04 dist-test-slave-wzcb krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:50:04.867890 13777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dee33443f0d4e4cbfaf6f75d654389d: Bootstrap starting.
I20260928 16:50:04.872824 13777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dee33443f0d4e4cbfaf6f75d654389d: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:04.874396 13777 log.cc:824] T 00000000000000000000000000000000 P 5dee33443f0d4e4cbfaf6f75d654389d: Log is configured to *not* fsync() on all Append() calls
I20260928 16:50:04.877787 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:04.852273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39450 (local address 127.9.93.62:42961)
0928 16:50:04.853198 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:04.853216 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:04.853267 (+ 51us) server_negotiation.cc:407] Connection header received
0928 16:50:04.855039 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:04.855066 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:04.855634 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:04.856091 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:04.858263 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:04.859976 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:04.861907 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:04.862705 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:04.862870 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:04.868077 (+ 5207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:04.868100 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:04.868124 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:04.868182 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:04.872279 (+ 4097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:04.873304 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:04.873315 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:04.873331 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:04.873606 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:04.874023 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:04.874028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:04.874032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:04.874904 (+ 872us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:04.875284 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:04.875778 (+ 494us) server_negotiation.cc:299] Negotiation successful
0928 16:50:04.876257 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":405,"threads_started":1}
I20260928 16:50:04.878710 13777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dee33443f0d4e4cbfaf6f75d654389d: No bootstrap required, opened a new log
I20260928 16:50:04.887173 13777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dee33443f0d4e4cbfaf6f75d654389d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dee33443f0d4e4cbfaf6f75d654389d" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42961 } }
I20260928 16:50:04.887826 13777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5dee33443f0d4e4cbfaf6f75d654389d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:04.887933 13777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dee33443f0d4e4cbfaf6f75d654389d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dee33443f0d4e4cbfaf6f75d654389d, State: Initialized, Role: FOLLOWER
I20260928 16:50:04.888607 13777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dee33443f0d4e4cbfaf6f75d654389d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dee33443f0d4e4cbfaf6f75d654389d" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42961 } }
I20260928 16:50:04.888849 13777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5dee33443f0d4e4cbfaf6f75d654389d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:50:04.888988 13777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dee33443f0d4e4cbfaf6f75d654389d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:50:04.889154 13777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dee33443f0d4e4cbfaf6f75d654389d [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:04.891556 13777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dee33443f0d4e4cbfaf6f75d654389d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dee33443f0d4e4cbfaf6f75d654389d" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42961 } }
I20260928 16:50:04.892185 13777 leader_election.cc:302] T 00000000000000000000000000000000 P 5dee33443f0d4e4cbfaf6f75d654389d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5dee33443f0d4e4cbfaf6f75d654389d; no voters:
I20260928 16:50:04.892800 13777 leader_election.cc:288] T 00000000000000000000000000000000 P 5dee33443f0d4e4cbfaf6f75d654389d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:50:04.892949 13783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dee33443f0d4e4cbfaf6f75d654389d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:50:04.893703 13783 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dee33443f0d4e4cbfaf6f75d654389d [term 1 LEADER]: Becoming Leader. State: Replica: 5dee33443f0d4e4cbfaf6f75d654389d, State: Running, Role: LEADER
I20260928 16:50:04.894313 13783 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dee33443f0d4e4cbfaf6f75d654389d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dee33443f0d4e4cbfaf6f75d654389d" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42961 } }
I20260928 16:50:04.895213 13777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dee33443f0d4e4cbfaf6f75d654389d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:50:04.897363 13785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dee33443f0d4e4cbfaf6f75d654389d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dee33443f0d4e4cbfaf6f75d654389d. Latest consensus state: current_term: 1 leader_uuid: "5dee33443f0d4e4cbfaf6f75d654389d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dee33443f0d4e4cbfaf6f75d654389d" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42961 } } }
I20260928 16:50:04.897930 13785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dee33443f0d4e4cbfaf6f75d654389d [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:04.897250 13784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dee33443f0d4e4cbfaf6f75d654389d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5dee33443f0d4e4cbfaf6f75d654389d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dee33443f0d4e4cbfaf6f75d654389d" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42961 } } }
I20260928 16:50:04.898860 13784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dee33443f0d4e4cbfaf6f75d654389d [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:04.900974 13789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:50:04.907934 13789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:50:04.919485 13789 catalog_manager.cc:1347] Generated new cluster ID: bb3c2648dd7f4811aff3a3b704942b4b
I20260928 16:50:04.919643 13789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:50:04.942751 13789 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:50:04.944981 13789 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:50:04.954654 13789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5dee33443f0d4e4cbfaf6f75d654389d: Generated new TSK 0
I20260928 16:50:04.955996 13789 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
I20260928 16:50:05.023671 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:42961
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:50:05.397130 13804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:05.397454 13804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:05.397567 13804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:05.405978 13804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:05.406124 13804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:05.406262 13804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:50:05.417362 13804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:42961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:05.418898 13804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:05.420531 13804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:05.435725 13812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:05.435819 13810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:05.435819 13809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:05.437649 13804 server_base.cc:1034] running on GCE node
I20260928 16:50:05.438423 13804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:05.439620 13804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:05.440848 13804 hybrid_clock.cc:648] HybridClock initialized: now 1790614205440783 us; error 59 us; skew 500 ppm
I20260928 16:50:05.441210 13804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:05 dist-test-slave-wzcb krb5kdc[13698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614205, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:05.444748 13804 init.cc:377] Logged in from keytab as kudu/127.9.93.1@KRBTEST.COM (short username kudu)
I20260928 16:50:05.446170 13804 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:05.446295 13804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:05.446667 13804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:05.450136 13804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "f08737a144734416a0fa6f5fd1d45c01"
format_stamp: "Formatted at 2026-09-28 16:50:05 on dist-test-slave-wzcb"
server_key: "bd5096e5f47327071745950daeb1cb32"
server_key_iv: "cbec7b307ab569a75ff0e5d3a841477d"
server_key_version: "encryptionkey@0"
I20260928 16:50:05.451074 13804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "f08737a144734416a0fa6f5fd1d45c01"
format_stamp: "Formatted at 2026-09-28 16:50:05 on dist-test-slave-wzcb"
server_key: "bd5096e5f47327071745950daeb1cb32"
server_key_iv: "cbec7b307ab569a75ff0e5d3a841477d"
server_key_version: "encryptionkey@0"
I20260928 16:50:05.457152 13804 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 16:50:05.461477 13818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:05.463353 13804 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 16:50:05.463563 13804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "f08737a144734416a0fa6f5fd1d45c01"
format_stamp: "Formatted at 2026-09-28 16:50:05 on dist-test-slave-wzcb"
server_key: "bd5096e5f47327071745950daeb1cb32"
server_key_iv: "cbec7b307ab569a75ff0e5d3a841477d"
server_key_version: "encryptionkey@0"
I20260928 16:50:05.463816 13804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:05.480249 13804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:05.484977 13804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:05.485287 13804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:05.486515 13804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:50:05.488235 13804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:05.488353 13804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:05.488473 13804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:05.488548 13804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:05.523828 13804 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:46399
I20260928 16:50:05.523895 13930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:46399 every 8 connection(s)
I20260928 16:50:05.524809 13804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:50:05.527132 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 13804
I20260928 16:50:05.527395 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:50:05.528263 9588 external_mini_cluster.cc:1428] Setting key 977abccfde590d2d3d6fbf27849be118
Sep 28 16:50:05 dist-test-slave-wzcb krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614205, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:50:05.550670 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:05.527753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:47723 (local address 127.9.93.62:42961)
0928 16:50:05.528323 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:05.528329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:05.529559 (+ 1230us) server_negotiation.cc:407] Connection header received
0928 16:50:05.532328 (+ 2769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:05.532335 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:05.532459 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:05.532591 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:05.535158 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:05.535946 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:05.538450 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:05.539050 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:05.539209 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:05.543472 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:05.543479 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:05.543482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:05.543533 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:05.546243 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:05.547490 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:05.547497 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:05.547499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:05.547678 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:05.548316 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:05.548321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:05.548324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:05.548717 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:05.548856 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:05.550000 (+ 1144us) server_negotiation.cc:299] Negotiation successful
0928 16:50:05.550305 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":273,"threads_started":1}
WARNING: no policy specified for kudu/127.9.93.2@KRBTEST.COM; defaulting to no policy
I20260928 16:50:05.552006 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:05.528410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42961 (local address 127.9.93.1:47723)
0928 16:50:05.529367 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:05.529433 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:05.531860 (+ 2427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:05.533026 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:05.533040 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:05.533943 (+ 903us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:05.534925 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:05.534956 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:05.536132 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:05.536137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:05.538201 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:05.538220 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:05.539375 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:05.539382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:05.539692 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:05.540411 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:05.540445 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:05.543125 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:05.546471 (+ 3346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:05.546483 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:05.546506 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:05.547329 (+ 823us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:05.547820 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:05.547826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:05.547830 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:05.548142 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:05.548877 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:05.548888 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:05.549456 (+ 568us) client_negotiation.cc:769] Sending connection context
0928 16:50:05.549842 (+ 386us) client_negotiation.cc:240] Negotiation successful
0928 16:50:05.550553 (+ 711us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":425,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.2@KRBTEST.COM" created.
I20260928 16:50:05.553855 13931 heartbeater.cc:347] Connected to a master server at 127.9.93.62:42961
I20260928 16:50:05.554280 13931 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:05.555373 13931 heartbeater.cc:511] Master 127.9.93.62:42961 requested a full tablet report, sending...
I20260928 16:50:05.557969 13742 ts_manager.cc:194] Registered new tserver with Master: f08737a144734416a0fa6f5fd1d45c01 (127.9.93.1:46399)
I20260928 16:50:05.559633 13742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.1@KRBTEST.COM'} at 127.9.93.1:47723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
I20260928 16:50:05.583071 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:42961
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:50:05.970834 13938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:05.971199 13938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:05.971403 13938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:05.979485 13938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:05.979640 13938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:05.979830 13938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:50:05.991040 13938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:42961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:05.992705 13938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:05.994277 13938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:06.009545 13943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:06.009485 13946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:06.009446 13944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:06.010646 13938 server_base.cc:1034] running on GCE node
I20260928 16:50:06.011436 13938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:06.012598 13938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:06.013813 13938 hybrid_clock.cc:648] HybridClock initialized: now 1790614206013740 us; error 66 us; skew 500 ppm
I20260928 16:50:06.014189 13938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:06 dist-test-slave-wzcb krb5kdc[13698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614206, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:06.017920 13938 init.cc:377] Logged in from keytab as kudu/127.9.93.2@KRBTEST.COM (short username kudu)
I20260928 16:50:06.019430 13938 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:06.019577 13938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:06.019994 13938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:06.023684 13938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "f8d63a2ae8b54c0b9c51e6e7fcb7420a"
format_stamp: "Formatted at 2026-09-28 16:50:06 on dist-test-slave-wzcb"
server_key: "0dc692e61c7c7ddf1c650df8824f32e1"
server_key_iv: "f491b2c7f72db49a2ca5468541ed7b81"
server_key_version: "encryptionkey@0"
I20260928 16:50:06.024715 13938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "f8d63a2ae8b54c0b9c51e6e7fcb7420a"
format_stamp: "Formatted at 2026-09-28 16:50:06 on dist-test-slave-wzcb"
server_key: "0dc692e61c7c7ddf1c650df8824f32e1"
server_key_iv: "f491b2c7f72db49a2ca5468541ed7b81"
server_key_version: "encryptionkey@0"
I20260928 16:50:06.031231 13938 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 16:50:06.036175 13952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:06.037997 13938 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:50:06.038223 13938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "f8d63a2ae8b54c0b9c51e6e7fcb7420a"
format_stamp: "Formatted at 2026-09-28 16:50:06 on dist-test-slave-wzcb"
server_key: "0dc692e61c7c7ddf1c650df8824f32e1"
server_key_iv: "f491b2c7f72db49a2ca5468541ed7b81"
server_key_version: "encryptionkey@0"
I20260928 16:50:06.038496 13938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:06.056443 13938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:06.061512 13938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:06.061836 13938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:06.063060 13938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:50:06.064775 13938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:06.064888 13938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:06.065016 13938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:06.065090 13938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:06.101327 13938 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:40227
I20260928 16:50:06.101369 14065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:40227 every 8 connection(s)
I20260928 16:50:06.102417 13938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:50:06.107659 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 13938
I20260928 16:50:06.107903 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:50:06.108490 9588 external_mini_cluster.cc:1428] Setting key 27ecb8cc365657f5364f27d2a86518cb
Sep 28 16:50:06 dist-test-slave-wzcb krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614206, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:50:06.129112 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:06.105493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:54879 (local address 127.9.93.62:42961)
0928 16:50:06.105990 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:06.105994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:06.107271 (+ 1277us) server_negotiation.cc:407] Connection header received
0928 16:50:06.110019 (+ 2748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:06.110024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:06.110093 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:06.110193 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:06.112711 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.113478 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:06.116049 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.116537 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:06.116653 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:06.121553 (+ 4900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:06.121562 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:06.121566 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:06.121612 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:06.124720 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:06.126042 (+ 1322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:06.126047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:06.126050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:06.126239 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:06.126831 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:06.126835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:06.126837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:06.127206 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:06.127407 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:06.128586 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0928 16:50:06.128809 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":253,"threads_started":1}
I20260928 16:50:06.131171 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:06.106177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42961 (local address 127.9.93.2:54879)
0928 16:50:06.107041 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:06.107090 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:06.109651 (+ 2561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:06.110552 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:06.110567 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:06.111430 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:06.112453 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:06.112494 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.113675 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:06.113681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:06.115821 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:06.115843 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.116823 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:06.116830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:06.117151 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:06.118006 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:06.118040 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:06.121136 (+ 3096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:06.124968 (+ 3832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:06.124980 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:06.125005 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:06.125838 (+ 833us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:06.126384 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:06.126390 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:06.126394 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:06.126707 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:06.127402 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:06.127413 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:06.128015 (+ 602us) client_negotiation.cc:769] Sending connection context
0928 16:50:06.128423 (+ 408us) client_negotiation.cc:240] Negotiation successful
0928 16:50:06.129201 (+ 778us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":353,"threads_started":1}
WARNING: no policy specified for kudu/127.9.93.3@KRBTEST.COM; defaulting to no policy
I20260928 16:50:06.133172 14066 heartbeater.cc:347] Connected to a master server at 127.9.93.62:42961
I20260928 16:50:06.133569 14066 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:06.134462 14066 heartbeater.cc:511] Master 127.9.93.62:42961 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.3@KRBTEST.COM" created.
I20260928 16:50:06.136613 13742 ts_manager.cc:194] Registered new tserver with Master: f8d63a2ae8b54c0b9c51e6e7fcb7420a (127.9.93.2:40227)
I20260928 16:50:06.137921 13742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.2@KRBTEST.COM'} at 127.9.93.2:54879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
I20260928 16:50:06.163252 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:42961
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:50:06.537827 14072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:06.538213 14072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:06.538398 14072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:06.546610 14072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:06.546775 14072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:06.546962 14072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:50:06.558096 14072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:42961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:06.559739 14072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:06.561332 14072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:50:06.563587 13931 heartbeater.cc:503] Master 127.9.93.62:42961 was elected leader, sending a full tablet report...
W20260928 16:50:06.576586 14078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:06.576665 14077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:06.576620 14080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:06.577610 14072 server_base.cc:1034] running on GCE node
I20260928 16:50:06.578380 14072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:06.579653 14072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:06.580888 14072 hybrid_clock.cc:648] HybridClock initialized: now 1790614206580822 us; error 51 us; skew 500 ppm
I20260928 16:50:06.581278 14072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:06 dist-test-slave-wzcb krb5kdc[13698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614206, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:06.584993 14072 init.cc:377] Logged in from keytab as kudu/127.9.93.3@KRBTEST.COM (short username kudu)
I20260928 16:50:06.586493 14072 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:06.586611 14072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:06.586979 14072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:06.590543 14072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "ff917f5096a845b3a04cb77508c6c9c1"
format_stamp: "Formatted at 2026-09-28 16:50:06 on dist-test-slave-wzcb"
server_key: "34fc86ce7cc0bdd780d11a7d67c0e712"
server_key_iv: "0bc2d18b3df473b3ad254fe5271973e2"
server_key_version: "encryptionkey@0"
I20260928 16:50:06.591574 14072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "ff917f5096a845b3a04cb77508c6c9c1"
format_stamp: "Formatted at 2026-09-28 16:50:06 on dist-test-slave-wzcb"
server_key: "34fc86ce7cc0bdd780d11a7d67c0e712"
server_key_iv: "0bc2d18b3df473b3ad254fe5271973e2"
server_key_version: "encryptionkey@0"
I20260928 16:50:06.597811 14072 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:50:06.602175 14086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:06.603931 14072 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:50:06.604136 14072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "ff917f5096a845b3a04cb77508c6c9c1"
format_stamp: "Formatted at 2026-09-28 16:50:06 on dist-test-slave-wzcb"
server_key: "34fc86ce7cc0bdd780d11a7d67c0e712"
server_key_iv: "0bc2d18b3df473b3ad254fe5271973e2"
server_key_version: "encryptionkey@0"
I20260928 16:50:06.604393 14072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:06.623082 14072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:06.627971 14072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:06.628288 14072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:06.629490 14072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:50:06.631285 14072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:06.631423 14072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260928 16:50:06.631559 14072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:06.631672 14072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:06.668258 14072 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:44649
I20260928 16:50:06.668299 14199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:44649 every 8 connection(s)
I20260928 16:50:06.669299 14072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:50:06.678164 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 14072
I20260928 16:50:06.678364 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:50:06.678817 9588 external_mini_cluster.cc:1428] Setting key 1ed6ace456ea97fdaafb30574deacd38
Sep 28 16:50:06 dist-test-slave-wzcb krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614206, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:50:06.693003 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:06.672176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:50005 (local address 127.9.93.62:42961)
0928 16:50:06.672664 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:06.672669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:06.673826 (+ 1157us) server_negotiation.cc:407] Connection header received
0928 16:50:06.676077 (+ 2251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:06.676084 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:06.676182 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:06.676292 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:06.678185 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.679087 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:06.681306 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.681776 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:06.681920 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:06.686046 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:06.686052 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:06.686055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:06.686102 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:06.689020 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:06.690120 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:06.690125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:06.690127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:06.690276 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:06.690838 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:06.690842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:06.690844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:06.691170 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:06.691334 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:06.692337 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0928 16:50:06.692637 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":247,"threads_started":1}
I20260928 16:50:06.694347 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:06.672779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42961 (local address 127.9.93.3:50005)
0928 16:50:06.673636 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:06.673684 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:06.675708 (+ 2024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:06.676600 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:06.676611 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:06.677262 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:06.677964 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:06.677984 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.679270 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:06.679274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:06.681130 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:06.681148 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.682085 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:06.682095 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:06.682400 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:06.683104 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:06.683146 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:06.685742 (+ 2596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:06.689257 (+ 3515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:06.689268 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:06.689287 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:06.689955 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:06.690403 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:06.690407 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:06.690410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:06.690712 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:06.691391 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:06.691400 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:06.691898 (+ 498us) client_negotiation.cc:769] Sending connection context
0928 16:50:06.692224 (+ 326us) client_negotiation.cc:240] Negotiation successful
0928 16:50:06.692783 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":348,"threads_started":1}
I20260928 16:50:06.696076 14200 heartbeater.cc:347] Connected to a master server at 127.9.93.62:42961
I20260928 16:50:06.696481 14200 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:06.697378 14200 heartbeater.cc:511] Master 127.9.93.62:42961 requested a full tablet report, sending...
I20260928 16:50:06.699155 13742 ts_manager.cc:194] Registered new tserver with Master: ff917f5096a845b3a04cb77508c6c9c1 (127.9.93.3:44649)
I20260928 16:50:06.700073 13742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.3@KRBTEST.COM'} at 127.9.93.3:50005
I20260928 16:50:06.708618 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:50:06.727742 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:06.713839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39452 (local address 127.9.93.62:42961)
0928 16:50:06.714121 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:06.714130 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:06.714611 (+ 481us) server_negotiation.cc:407] Connection header received
0928 16:50:06.714820 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:06.714826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:06.714915 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:06.715030 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:06.716679 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.717465 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:06.718896 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.719399 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:06.719509 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:06.722154 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:06.722160 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:06.722164 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:06.722209 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:06.725063 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:06.725831 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:06.725835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:06.725838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:06.725977 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:06.726331 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:06.726334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:06.726337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:06.726703 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:06.726915 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:06.727219 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 16:50:06.727467 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 16:50:06.732260 13742 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:50:06.739862 13742 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:50:06.767653 14213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:06.754218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:40227 (local address 127.0.0.1:44686)
0928 16:50:06.755088 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:06.755143 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:06.755369 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:06.756747 (+ 1378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:06.756755 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:06.756792 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:50:06.757227 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:06.757242 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.761353 (+ 4111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:06.761358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:06.764084 (+ 2726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:06.764111 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.766210 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:06.766217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:06.766546 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:06.766634 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 16:50:06.766791 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 16:50:06.767047 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":57,"thread_start_us":820,"threads_started":1}
I20260928 16:50:06.769266 14215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:06.754807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44686 (local address 127.9.93.2:40227)
0928 16:50:06.755954 (+ 1147us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:06.755965 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:06.755998 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:50:06.756115 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:06.756126 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:06.756453 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:06.757487 (+ 1034us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:50:06.757755 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.759493 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:06.764299 (+ 4806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.765958 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:06.766131 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:06.766375 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:06.768641 (+ 2266us) server_negotiation.cc:299] Negotiation successful
0928 16:50:06.768885 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":954,"thread_start_us":1935,"threads_started":1}
I20260928 16:50:06.769573 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:06.754352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:46399 (local address 127.0.0.1:39340)
0928 16:50:06.760114 (+ 5762us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:06.760137 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:06.760272 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:06.761656 (+ 1384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:06.761660 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:06.761686 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:50:06.762238 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:06.762253 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.764337 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:06.764342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:06.766208 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:06.766223 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.768670 (+ 2447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:06.768676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:06.768870 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:06.768908 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 16:50:06.769001 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 16:50:06.769174 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5610,"mutex_wait_us":3,"thread_start_us":5451,"threads_started":1}
I20260928 16:50:06.769935 14218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:06.756580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.9.93.3:44649)
0928 16:50:06.761199 (+ 4619us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:06.761209 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:06.761341 (+ 132us) server_negotiation.cc:407] Connection header received
0928 16:50:06.761536 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:06.761544 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:06.761840 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:06.762083 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:50:06.762876 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.764559 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:06.767209 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.768707 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:06.768844 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:06.769114 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:06.769341 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 16:50:06.769536 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4397,"thread_start_us":4203,"threads_started":1}
I20260928 16:50:06.769963 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:06.756005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:44649 (local address 127.0.0.1:59436)
0928 16:50:06.761217 (+ 5212us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:06.761237 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:06.761377 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:06.762053 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:06.762057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:06.762092 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:50:06.762611 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:06.762624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.764738 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:06.764743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:06.766964 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:06.766979 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.768932 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:06.768938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:06.769134 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:06.769170 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 16:50:06.769376 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 16:50:06.769536 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5077,"mutex_wait_us":27,"thread_start_us":5068,"threads_started":1}
I20260928 16:50:06.771667 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:06.754993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39340 (local address 127.9.93.1:46399)
0928 16:50:06.760922 (+ 5929us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:06.760933 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:06.760965 (+ 32us) server_negotiation.cc:407] Connection header received
0928 16:50:06.761076 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:06.761086 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:06.761411 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:06.761675 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:50:06.762419 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.764189 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:06.766426 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.768063 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:06.768226 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:06.768518 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:06.771049 (+ 2531us) server_negotiation.cc:299] Negotiation successful
0928 16:50:06.771251 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5715,"thread_start_us":5517,"threads_started":1}
I20260928 16:50:06.774411 14001 tablet_service.cc:1467] Processing CreateTablet for tablet d0b549177be1457391d230ff02c61572 (DEFAULT_TABLE table=test-table [id=3c79c67876474723a195ca6d7a94584c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:50:06.775189 13866 tablet_service.cc:1467] Processing CreateTablet for tablet d0b549177be1457391d230ff02c61572 (DEFAULT_TABLE table=test-table [id=3c79c67876474723a195ca6d7a94584c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:50:06.775266 14135 tablet_service.cc:1467] Processing CreateTablet for tablet d0b549177be1457391d230ff02c61572 (DEFAULT_TABLE table=test-table [id=3c79c67876474723a195ca6d7a94584c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:50:06.776257 14001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b549177be1457391d230ff02c61572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:06.776527 13866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b549177be1457391d230ff02c61572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:06.776780 14135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b549177be1457391d230ff02c61572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:06.790035 14219 tablet_bootstrap.cc:492] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01: Bootstrap starting.
I20260928 16:50:06.793630 14221 tablet_bootstrap.cc:492] T d0b549177be1457391d230ff02c61572 P ff917f5096a845b3a04cb77508c6c9c1: Bootstrap starting.
I20260928 16:50:06.795766 14219 tablet_bootstrap.cc:654] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:06.797447 14219 log.cc:824] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01: Log is configured to *not* fsync() on all Append() calls
I20260928 16:50:06.797940 14221 tablet_bootstrap.cc:654] T d0b549177be1457391d230ff02c61572 P ff917f5096a845b3a04cb77508c6c9c1: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:06.799350 14220 tablet_bootstrap.cc:492] T d0b549177be1457391d230ff02c61572 P f8d63a2ae8b54c0b9c51e6e7fcb7420a: Bootstrap starting.
I20260928 16:50:06.799579 14221 log.cc:824] T d0b549177be1457391d230ff02c61572 P ff917f5096a845b3a04cb77508c6c9c1: Log is configured to *not* fsync() on all Append() calls
I20260928 16:50:06.801757 14219 tablet_bootstrap.cc:492] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01: No bootstrap required, opened a new log
I20260928 16:50:06.802074 14219 ts_tablet_manager.cc:1397] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260928 16:50:06.804986 14220 tablet_bootstrap.cc:654] T d0b549177be1457391d230ff02c61572 P f8d63a2ae8b54c0b9c51e6e7fcb7420a: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:06.806598 14220 log.cc:824] T d0b549177be1457391d230ff02c61572 P f8d63a2ae8b54c0b9c51e6e7fcb7420a: Log is configured to *not* fsync() on all Append() calls
I20260928 16:50:06.807402 14221 tablet_bootstrap.cc:492] T d0b549177be1457391d230ff02c61572 P ff917f5096a845b3a04cb77508c6c9c1: No bootstrap required, opened a new log
I20260928 16:50:06.807806 14221 ts_tablet_manager.cc:1397] T d0b549177be1457391d230ff02c61572 P ff917f5096a845b3a04cb77508c6c9c1: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.004s
I20260928 16:50:06.810573 14220 tablet_bootstrap.cc:492] T d0b549177be1457391d230ff02c61572 P f8d63a2ae8b54c0b9c51e6e7fcb7420a: No bootstrap required, opened a new log
I20260928 16:50:06.810930 14220 ts_tablet_manager.cc:1397] T d0b549177be1457391d230ff02c61572 P f8d63a2ae8b54c0b9c51e6e7fcb7420a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.005s
I20260928 16:50:06.811470 14219 raft_consensus.cc:348] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d63a2ae8b54c0b9c51e6e7fcb7420a" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 40227 } } peers { permanent_uuid: "f08737a144734416a0fa6f5fd1d45c01" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 46399 } } peers { permanent_uuid: "ff917f5096a845b3a04cb77508c6c9c1" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44649 } }
I20260928 16:50:06.811926 14219 raft_consensus.cc:374] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:06.812100 14219 raft_consensus.cc:729] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f08737a144734416a0fa6f5fd1d45c01, State: Initialized, Role: FOLLOWER
I20260928 16:50:06.812937 14219 consensus_queue.cc:260] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d63a2ae8b54c0b9c51e6e7fcb7420a" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 40227 } } peers { permanent_uuid: "f08737a144734416a0fa6f5fd1d45c01" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 46399 } } peers { permanent_uuid: "ff917f5096a845b3a04cb77508c6c9c1" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44649 } }
I20260928 16:50:06.820966 14219 ts_tablet_manager.cc:1428] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01: Time spent starting tablet: real 0.019s user 0.013s sys 0.004s
I20260928 16:50:06.819792 14221 raft_consensus.cc:348] T d0b549177be1457391d230ff02c61572 P ff917f5096a845b3a04cb77508c6c9c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d63a2ae8b54c0b9c51e6e7fcb7420a" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 40227 } } peers { permanent_uuid: "f08737a144734416a0fa6f5fd1d45c01" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 46399 } } peers { permanent_uuid: "ff917f5096a845b3a04cb77508c6c9c1" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44649 } }
I20260928 16:50:06.821606 14221 raft_consensus.cc:374] T d0b549177be1457391d230ff02c61572 P ff917f5096a845b3a04cb77508c6c9c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:06.821743 14221 raft_consensus.cc:729] T d0b549177be1457391d230ff02c61572 P ff917f5096a845b3a04cb77508c6c9c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff917f5096a845b3a04cb77508c6c9c1, State: Initialized, Role: FOLLOWER
I20260928 16:50:06.822531 14221 consensus_queue.cc:260] T d0b549177be1457391d230ff02c61572 P ff917f5096a845b3a04cb77508c6c9c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d63a2ae8b54c0b9c51e6e7fcb7420a" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 40227 } } peers { permanent_uuid: "f08737a144734416a0fa6f5fd1d45c01" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 46399 } } peers { permanent_uuid: "ff917f5096a845b3a04cb77508c6c9c1" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44649 } }
I20260928 16:50:06.822316 14220 raft_consensus.cc:348] T d0b549177be1457391d230ff02c61572 P f8d63a2ae8b54c0b9c51e6e7fcb7420a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d63a2ae8b54c0b9c51e6e7fcb7420a" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 40227 } } peers { permanent_uuid: "f08737a144734416a0fa6f5fd1d45c01" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 46399 } } peers { permanent_uuid: "ff917f5096a845b3a04cb77508c6c9c1" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44649 } }
I20260928 16:50:06.822883 14220 raft_consensus.cc:374] T d0b549177be1457391d230ff02c61572 P f8d63a2ae8b54c0b9c51e6e7fcb7420a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:06.823010 14220 raft_consensus.cc:729] T d0b549177be1457391d230ff02c61572 P f8d63a2ae8b54c0b9c51e6e7fcb7420a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8d63a2ae8b54c0b9c51e6e7fcb7420a, State: Initialized, Role: FOLLOWER
I20260928 16:50:06.823755 14220 consensus_queue.cc:260] T d0b549177be1457391d230ff02c61572 P f8d63a2ae8b54c0b9c51e6e7fcb7420a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d63a2ae8b54c0b9c51e6e7fcb7420a" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 40227 } } peers { permanent_uuid: "f08737a144734416a0fa6f5fd1d45c01" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 46399 } } peers { permanent_uuid: "ff917f5096a845b3a04cb77508c6c9c1" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44649 } }
I20260928 16:50:06.826290 14200 heartbeater.cc:503] Master 127.9.93.62:42961 was elected leader, sending a full tablet report...
I20260928 16:50:06.827814 14221 ts_tablet_manager.cc:1428] T d0b549177be1457391d230ff02c61572 P ff917f5096a845b3a04cb77508c6c9c1: Time spent starting tablet: real 0.020s user 0.010s sys 0.008s
I20260928 16:50:06.830049 14066 heartbeater.cc:503] Master 127.9.93.62:42961 was elected leader, sending a full tablet report...
I20260928 16:50:06.830787 14220 ts_tablet_manager.cc:1428] T d0b549177be1457391d230ff02c61572 P f8d63a2ae8b54c0b9c51e6e7fcb7420a: Time spent starting tablet: real 0.020s user 0.007s sys 0.011s
W20260928 16:50:06.854573 14067 tablet.cc:2369] T d0b549177be1457391d230ff02c61572 P f8d63a2ae8b54c0b9c51e6e7fcb7420a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:50:06.920966 14201 tablet.cc:2369] T d0b549177be1457391d230ff02c61572 P ff917f5096a845b3a04cb77508c6c9c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:50:06.992697 14225 raft_consensus.cc:482] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:50:06.993081 14225 raft_consensus.cc:504] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d63a2ae8b54c0b9c51e6e7fcb7420a" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 40227 } } peers { permanent_uuid: "f08737a144734416a0fa6f5fd1d45c01" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 46399 } } peers { permanent_uuid: "ff917f5096a845b3a04cb77508c6c9c1" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44649 } }
I20260928 16:50:06.995138 14225 leader_election.cc:288] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8d63a2ae8b54c0b9c51e6e7fcb7420a (127.9.93.2:40227), ff917f5096a845b3a04cb77508c6c9c1 (127.9.93.3:44649)
I20260928 16:50:07.004390 14228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:06.995588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:40227 (local address 127.9.93.1:46769)
0928 16:50:06.996280 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:06.996383 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:06.996550 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:06.997053 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:06.997057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:06.997085 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:50:06.997537 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:06.997550 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.998885 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:06.998888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:07.001376 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:07.001396 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:07.003343 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:07.003351 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:07.003626 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:07.003668 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 16:50:07.003759 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 16:50:07.003924 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"spinlock_wait_cycles":27392,"thread_start_us":329,"threads_started":1}
I20260928 16:50:07.004436 14215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:06.995807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:46769 (local address 127.9.93.2:40227)
0928 16:50:06.996047 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:06.996052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:06.996495 (+ 443us) server_negotiation.cc:407] Connection header received
0928 16:50:06.996748 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:06.996766 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:06.996876 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:06.996993 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:50:06.997713 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.998743 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:07.001599 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:07.003042 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:07.003230 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:07.003379 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:07.003817 (+ 438us) server_negotiation.cc:299] Negotiation successful
0928 16:50:07.003971 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 16:50:07.004618 14218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:06.996804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:55723 (local address 127.9.93.3:44649)
0928 16:50:06.997057 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:06.997063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:06.997229 (+ 166us) server_negotiation.cc:407] Connection header received
0928 16:50:06.997418 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:06.997424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:06.997496 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:06.997573 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:50:06.998149 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.999107 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:07.001802 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:07.003168 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:07.003286 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:07.003411 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:07.004122 (+ 711us) server_negotiation.cc:299] Negotiation successful
0928 16:50:07.004282 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260928 16:50:07.005111 14230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:06.996485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:44649 (local address 127.9.93.1:55723)
0928 16:50:06.997125 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:06.997145 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:06.997296 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:06.997612 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:06.997617 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:06.997643 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:50:06.997985 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:06.997993 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:06.999235 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:06.999241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:07.001613 (+ 2372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:07.001625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:07.003687 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:07.003694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:07.003896 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:07.003929 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 16:50:07.004634 (+ 705us) client_negotiation.cc:240] Negotiation successful
0928 16:50:07.004784 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":315,"threads_started":1}
I20260928 16:50:07.005733 14021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b549177be1457391d230ff02c61572" candidate_uuid: "f08737a144734416a0fa6f5fd1d45c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8d63a2ae8b54c0b9c51e6e7fcb7420a" is_pre_election: true
I20260928 16:50:07.006146 14155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b549177be1457391d230ff02c61572" candidate_uuid: "f08737a144734416a0fa6f5fd1d45c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff917f5096a845b3a04cb77508c6c9c1" is_pre_election: true
I20260928 16:50:07.006297 14021 raft_consensus.cc:2454] T d0b549177be1457391d230ff02c61572 P f8d63a2ae8b54c0b9c51e6e7fcb7420a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f08737a144734416a0fa6f5fd1d45c01 in term 0.
I20260928 16:50:07.006669 14155 raft_consensus.cc:2454] T d0b549177be1457391d230ff02c61572 P ff917f5096a845b3a04cb77508c6c9c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f08737a144734416a0fa6f5fd1d45c01 in term 0.
I20260928 16:50:07.007113 13821 leader_election.cc:302] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f08737a144734416a0fa6f5fd1d45c01, f8d63a2ae8b54c0b9c51e6e7fcb7420a; no voters:
I20260928 16:50:07.007616 14225 raft_consensus.cc:2781] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:50:07.007773 14225 raft_consensus.cc:482] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:50:07.007885 14225 raft_consensus.cc:3037] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:07.010571 14225 raft_consensus.cc:504] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d63a2ae8b54c0b9c51e6e7fcb7420a" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 40227 } } peers { permanent_uuid: "f08737a144734416a0fa6f5fd1d45c01" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 46399 } } peers { permanent_uuid: "ff917f5096a845b3a04cb77508c6c9c1" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44649 } }
I20260928 16:50:07.011344 14225 leader_election.cc:288] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01 [CANDIDATE]: Term 1 election: Requested vote from peers f8d63a2ae8b54c0b9c51e6e7fcb7420a (127.9.93.2:40227), ff917f5096a845b3a04cb77508c6c9c1 (127.9.93.3:44649)
I20260928 16:50:07.012070 14021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b549177be1457391d230ff02c61572" candidate_uuid: "f08737a144734416a0fa6f5fd1d45c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8d63a2ae8b54c0b9c51e6e7fcb7420a"
I20260928 16:50:07.012094 14155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b549177be1457391d230ff02c61572" candidate_uuid: "f08737a144734416a0fa6f5fd1d45c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff917f5096a845b3a04cb77508c6c9c1"
I20260928 16:50:07.012322 14021 raft_consensus.cc:3037] T d0b549177be1457391d230ff02c61572 P f8d63a2ae8b54c0b9c51e6e7fcb7420a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:07.012319 14155 raft_consensus.cc:3037] T d0b549177be1457391d230ff02c61572 P ff917f5096a845b3a04cb77508c6c9c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:07.014925 14155 raft_consensus.cc:2454] T d0b549177be1457391d230ff02c61572 P ff917f5096a845b3a04cb77508c6c9c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f08737a144734416a0fa6f5fd1d45c01 in term 1.
I20260928 16:50:07.014914 14021 raft_consensus.cc:2454] T d0b549177be1457391d230ff02c61572 P f8d63a2ae8b54c0b9c51e6e7fcb7420a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f08737a144734416a0fa6f5fd1d45c01 in term 1.
I20260928 16:50:07.015501 13821 leader_election.cc:302] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f08737a144734416a0fa6f5fd1d45c01, ff917f5096a845b3a04cb77508c6c9c1; no voters:
I20260928 16:50:07.015874 14225 raft_consensus.cc:2781] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:50:07.016463 14225 raft_consensus.cc:686] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01 [term 1 LEADER]: Becoming Leader. State: Replica: f08737a144734416a0fa6f5fd1d45c01, State: Running, Role: LEADER
I20260928 16:50:07.017122 14225 consensus_queue.cc:237] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d63a2ae8b54c0b9c51e6e7fcb7420a" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 40227 } } peers { permanent_uuid: "f08737a144734416a0fa6f5fd1d45c01" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 46399 } } peers { permanent_uuid: "ff917f5096a845b3a04cb77508c6c9c1" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44649 } }
I20260928 16:50:07.022331 13742 catalog_manager.cc:5495] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01 reported cstate change: term changed from 0 to 1, leader changed from <none> to f08737a144734416a0fa6f5fd1d45c01 (127.9.93.1). New cstate: current_term: 1 leader_uuid: "f08737a144734416a0fa6f5fd1d45c01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d63a2ae8b54c0b9c51e6e7fcb7420a" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 40227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f08737a144734416a0fa6f5fd1d45c01" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 46399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff917f5096a845b3a04cb77508c6c9c1" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44649 } health_report { overall_health: UNKNOWN } } }
W20260928 16:50:07.027581 13932 tablet.cc:2369] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:50:07.076241 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:07.070299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39350 (local address 127.9.93.1:46399)
0928 16:50:07.070495 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:07.070500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:07.070547 (+ 47us) server_negotiation.cc:407] Connection header received
0928 16:50:07.070820 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:07.070824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:07.070934 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:07.071079 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:50:07.071617 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:07.072401 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:07.073931 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:07.074384 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:07.074491 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:07.074985 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:50:07.075674 (+ 689us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:50:07.075799 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:07.075912 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 16:50:07.076008 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:50:07.086688 14021 raft_consensus.cc:1261] T d0b549177be1457391d230ff02c61572 P f8d63a2ae8b54c0b9c51e6e7fcb7420a [term 1 FOLLOWER]: Refusing update from remote peer f08737a144734416a0fa6f5fd1d45c01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:50:07.086684 14155 raft_consensus.cc:1261] T d0b549177be1457391d230ff02c61572 P ff917f5096a845b3a04cb77508c6c9c1 [term 1 FOLLOWER]: Refusing update from remote peer f08737a144734416a0fa6f5fd1d45c01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:50:07.087976 14231 consensus_queue.cc:1035] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff917f5096a845b3a04cb77508c6c9c1" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:50:07.088485 14225 consensus_queue.cc:1035] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8d63a2ae8b54c0b9c51e6e7fcb7420a" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 40227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:50:07.111492 14235 mvcc.cc:205] Tried to move back new op lower bound from 7334355792213651456 to 7334355791946682368. Current Snapshot: MvccSnapshot[applied={T|T < 7334355792213651456}]
I20260928 16:50:07.133745 13742 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:50:07.134116 13742 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:50:07.138636 13742 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5dee33443f0d4e4cbfaf6f75d654389d: Sending DeleteTablet for 3 replicas of tablet d0b549177be1457391d230ff02c61572
I20260928 16:50:07.140105 13866 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b549177be1457391d230ff02c61572 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:07 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:39340
I20260928 16:50:07.140202 14135 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b549177be1457391d230ff02c61572 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:07 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:59436
I20260928 16:50:07.140202 14001 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b549177be1457391d230ff02c61572 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:07 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:44686
I20260928 16:50:07.143014 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 13804
I20260928 16:50:07.153257 14245 tablet_replica.cc:330] T d0b549177be1457391d230ff02c61572 P f8d63a2ae8b54c0b9c51e6e7fcb7420a: stopping tablet replica
I20260928 16:50:07.154217 14245 raft_consensus.cc:2229] T d0b549177be1457391d230ff02c61572 P f8d63a2ae8b54c0b9c51e6e7fcb7420a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:50:07.155261 14245 raft_consensus.cc:2258] T d0b549177be1457391d230ff02c61572 P f8d63a2ae8b54c0b9c51e6e7fcb7420a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:07.156764 14244 tablet_replica.cc:330] T d0b549177be1457391d230ff02c61572 P ff917f5096a845b3a04cb77508c6c9c1: stopping tablet replica
I20260928 16:50:07.158609 14246 tablet_replica.cc:330] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01: stopping tablet replica
I20260928 16:50:07.158805 14245 ts_tablet_manager.cc:1905] T d0b549177be1457391d230ff02c61572 P f8d63a2ae8b54c0b9c51e6e7fcb7420a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:50:07.159149 14246 raft_consensus.cc:2229] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:50:07.159535 14244 raft_consensus.cc:2229] T d0b549177be1457391d230ff02c61572 P ff917f5096a845b3a04cb77508c6c9c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:50:07.160175 14246 raft_consensus.cc:2258] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:07.160255 14244 raft_consensus.cc:2258] T d0b549177be1457391d230ff02c61572 P ff917f5096a845b3a04cb77508c6c9c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:07.162943 14246 ts_tablet_manager.cc:1905] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:50:07.166088 14244 ts_tablet_manager.cc:1905] T d0b549177be1457391d230ff02c61572 P ff917f5096a845b3a04cb77508c6c9c1: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 16:50:07 dist-test-slave-wzcb krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.1@KRBTEST.COM
I20260928 16:50:07.172235 14246 ts_tablet_manager.cc:1918] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:50:07.172592 14246 log.cc:1196] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-0/wal/wals/d0b549177be1457391d230ff02c61572
I20260928 16:50:07.173436 14246 ts_tablet_manager.cc:1939] T d0b549177be1457391d230ff02c61572 P f08737a144734416a0fa6f5fd1d45c01: Deleting consensus metadata
I20260928 16:50:07.175968 13729 catalog_manager.cc:4841] TS f08737a144734416a0fa6f5fd1d45c01 (127.9.93.1:46399): tablet d0b549177be1457391d230ff02c61572 (table test-table [id=3c79c67876474723a195ca6d7a94584c]) successfully deleted
I20260928 16:50:07.177290 14245 ts_tablet_manager.cc:1918] T d0b549177be1457391d230ff02c61572 P f8d63a2ae8b54c0b9c51e6e7fcb7420a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:50:07.177650 14245 log.cc:1196] T d0b549177be1457391d230ff02c61572 P f8d63a2ae8b54c0b9c51e6e7fcb7420a: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/d0b549177be1457391d230ff02c61572
I20260928 16:50:07.178372 14245 ts_tablet_manager.cc:1939] T d0b549177be1457391d230ff02c61572 P f8d63a2ae8b54c0b9c51e6e7fcb7420a: Deleting consensus metadata
I20260928 16:50:07.179677 14244 ts_tablet_manager.cc:1918] T d0b549177be1457391d230ff02c61572 P ff917f5096a845b3a04cb77508c6c9c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:50:07.179961 14244 log.cc:1196] T d0b549177be1457391d230ff02c61572 P ff917f5096a845b3a04cb77508c6c9c1: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/d0b549177be1457391d230ff02c61572
I20260928 16:50:07.180272 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:07.145053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39352 (local address 127.9.93.1:46399)
0928 16:50:07.145521 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:07.145529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:07.157072 (+ 11543us) server_negotiation.cc:407] Connection header received
0928 16:50:07.158910 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:07.158917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:07.159048 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:07.159143 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:07.161127 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:07.162300 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:07.164182 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:07.164926 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:07.165033 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:07.171421 (+ 6388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:07.171441 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:07.171462 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:07.171518 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:07.176091 (+ 4573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:07.177182 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:07.177193 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:07.177197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:07.177421 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:07.177989 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:07.177994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:07.177997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:07.178550 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:07.178866 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:07.179561 (+ 695us) server_negotiation.cc:299] Negotiation successful
0928 16:50:07.179884 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316}
I20260928 16:50:07.180871 13728 catalog_manager.cc:4841] TS f8d63a2ae8b54c0b9c51e6e7fcb7420a (127.9.93.2:40227): tablet d0b549177be1457391d230ff02c61572 (table test-table [id=3c79c67876474723a195ca6d7a94584c]) successfully deleted
I20260928 16:50:07.181293 14244 ts_tablet_manager.cc:1939] T d0b549177be1457391d230ff02c61572 P ff917f5096a845b3a04cb77508c6c9c1: Deleting consensus metadata
I20260928 16:50:07.182013 13926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:07.183382 13728 catalog_manager.cc:4841] TS ff917f5096a845b3a04cb77508c6c9c1 (127.9.93.3:44649): tablet d0b549177be1457391d230ff02c61572 (table test-table [id=3c79c67876474723a195ca6d7a94584c]) successfully deleted
I20260928 16:50:07.289640 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 13804
I20260928 16:50:07.307096 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 13938
Sep 28 16:50:07 dist-test-slave-wzcb krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.2@KRBTEST.COM
I20260928 16:50:07.323873 14215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:07.308214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44702 (local address 127.9.93.2:40227)
0928 16:50:07.308445 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:07.308450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:07.308473 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:50:07.308552 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:07.308559 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:07.308702 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:07.308822 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:07.310266 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:07.311049 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:07.312455 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:07.312911 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:07.313041 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:07.317422 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:07.317442 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:07.317460 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:07.317509 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:07.320626 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:07.321474 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:07.321483 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:07.321486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:07.321659 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:07.322077 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:07.322081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:07.322084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:07.322537 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:07.322871 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:07.323209 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 16:50:07.323510 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 16:50:07.324626 14061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:07.421001 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 13938
I20260928 16:50:07.436721 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 14072
Sep 28 16:50:07 dist-test-slave-wzcb krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.3@KRBTEST.COM
I20260928 16:50:07.454392 14218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:07.437722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59442 (local address 127.9.93.3:44649)
0928 16:50:07.437936 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:07.437943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:07.437969 (+ 26us) server_negotiation.cc:407] Connection header received
0928 16:50:07.438124 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:07.438130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:07.438218 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:07.438317 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:07.439836 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:07.440734 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:07.442164 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:07.442702 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:07.442842 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:07.448141 (+ 5299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:07.448162 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:07.448182 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:07.448233 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:07.451182 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:07.452069 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:07.452079 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:07.452081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:07.452253 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:07.452688 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:07.452692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:07.452694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:07.453125 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:07.453451 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:07.453777 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 16:50:07.454033 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 16:50:07.455178 14195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:07.558275 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 14072
I20260928 16:50:07.576526 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 13711
I20260928 16:50:07.577812 13772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:07.694504 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 13711
2026-09-28T16:50:07Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4562 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T16:50:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:50:07Z Disabled control of system clock
I20260928 16:50:07.766402 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:34401
--webserver_interface=127.9.93.62
--webserver_port=0
--builtin_ntp_servers=127.9.93.20:43709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:34401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 16:50:08.131178 14255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:08.131582 14255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:08.131722 14255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:08.139639 14255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:50:08.139801 14255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:08.139870 14255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:08.139961 14255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:50:08.140026 14255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:50:08.150472 14255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:43709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:34401
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:34401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:08.152099 14255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:08.154147 14255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:08.167840 14263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:08.167768 14260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:08.167797 14261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:08.168843 14255 server_base.cc:1034] running on GCE node
I20260928 16:50:08.169859 14255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:08.171659 14255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:08.172924 14255 hybrid_clock.cc:648] HybridClock initialized: now 1790614208172854 us; error 49 us; skew 500 ppm
I20260928 16:50:08.173352 14255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:50:08.176018 14255 webserver.cc:460] Webserver started at http://127.9.93.62:33629/ using document root <none> and password file <none>
I20260928 16:50:08.176940 14255 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:08.177057 14255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:08.177403 14255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:08.181147 14255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "dd29acbf656c48cd8d9cfd1fa99bcf38"
format_stamp: "Formatted at 2026-09-28 16:50:08 on dist-test-slave-wzcb"
server_key: "885116722107c3b147f6d14ca4428efa"
server_key_iv: "d775fdc5ddb746e9cdd146a1e90f97a7"
server_key_version: "encryptionkey@0"
I20260928 16:50:08.182212 14255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "dd29acbf656c48cd8d9cfd1fa99bcf38"
format_stamp: "Formatted at 2026-09-28 16:50:08 on dist-test-slave-wzcb"
server_key: "885116722107c3b147f6d14ca4428efa"
server_key_iv: "d775fdc5ddb746e9cdd146a1e90f97a7"
server_key_version: "encryptionkey@0"
I20260928 16:50:08.188580 14255 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 16:50:08.192754 14269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:08.194680 14255 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:50:08.194909 14255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "dd29acbf656c48cd8d9cfd1fa99bcf38"
format_stamp: "Formatted at 2026-09-28 16:50:08 on dist-test-slave-wzcb"
server_key: "885116722107c3b147f6d14ca4428efa"
server_key_iv: "d775fdc5ddb746e9cdd146a1e90f97a7"
server_key_version: "encryptionkey@0"
I20260928 16:50:08.195179 14255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:08.201920 14255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:08.202901 14255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:08.203233 14255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:08.222259 14255 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:34401
I20260928 16:50:08.222273 14320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:34401 every 8 connection(s)
I20260928 16:50:08.223909 14255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:50:08.228307 14321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:08.230472 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 14255
I20260928 16:50:08.230690 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:50:08.231637 9588 external_mini_cluster.cc:1428] Setting key a27b3c580b2de99b6ddcfb668e68a4d0
I20260928 16:50:08.242161 14321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd29acbf656c48cd8d9cfd1fa99bcf38: Bootstrap starting.
I20260928 16:50:08.246200 14321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd29acbf656c48cd8d9cfd1fa99bcf38: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:08.247589 14321 log.cc:824] T 00000000000000000000000000000000 P dd29acbf656c48cd8d9cfd1fa99bcf38: Log is configured to *not* fsync() on all Append() calls
I20260928 16:50:08.248003 14324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:08.234209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45644 (local address 127.9.93.62:34401)
0928 16:50:08.235509 (+ 1300us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:08.235525 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:08.235592 (+ 67us) server_negotiation.cc:407] Connection header received
0928 16:50:08.237503 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:08.237542 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:08.238086 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:08.238553 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:08.245743 (+ 7190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:08.245752 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:08.245771 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:50:08.245796 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:08.245890 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:50:08.245911 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:08.246137 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:08.246327 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 16:50:08.246752 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":425,"threads_started":1}
I20260928 16:50:08.251073 14321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd29acbf656c48cd8d9cfd1fa99bcf38: No bootstrap required, opened a new log
I20260928 16:50:08.259553 14321 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd29acbf656c48cd8d9cfd1fa99bcf38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd29acbf656c48cd8d9cfd1fa99bcf38" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 34401 } }
I20260928 16:50:08.260003 14321 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd29acbf656c48cd8d9cfd1fa99bcf38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:08.260095 14321 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd29acbf656c48cd8d9cfd1fa99bcf38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd29acbf656c48cd8d9cfd1fa99bcf38, State: Initialized, Role: FOLLOWER
I20260928 16:50:08.260725 14321 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd29acbf656c48cd8d9cfd1fa99bcf38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd29acbf656c48cd8d9cfd1fa99bcf38" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 34401 } }
I20260928 16:50:08.260979 14321 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd29acbf656c48cd8d9cfd1fa99bcf38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:50:08.261096 14321 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd29acbf656c48cd8d9cfd1fa99bcf38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:50:08.261236 14321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd29acbf656c48cd8d9cfd1fa99bcf38 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:08.263890 14321 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd29acbf656c48cd8d9cfd1fa99bcf38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd29acbf656c48cd8d9cfd1fa99bcf38" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 34401 } }
I20260928 16:50:08.264549 14321 leader_election.cc:302] T 00000000000000000000000000000000 P dd29acbf656c48cd8d9cfd1fa99bcf38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd29acbf656c48cd8d9cfd1fa99bcf38; no voters:
I20260928 16:50:08.265196 14321 leader_election.cc:288] T 00000000000000000000000000000000 P dd29acbf656c48cd8d9cfd1fa99bcf38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:50:08.265373 14326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd29acbf656c48cd8d9cfd1fa99bcf38 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:50:08.266110 14326 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd29acbf656c48cd8d9cfd1fa99bcf38 [term 1 LEADER]: Becoming Leader. State: Replica: dd29acbf656c48cd8d9cfd1fa99bcf38, State: Running, Role: LEADER
I20260928 16:50:08.266741 14326 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd29acbf656c48cd8d9cfd1fa99bcf38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd29acbf656c48cd8d9cfd1fa99bcf38" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 34401 } }
I20260928 16:50:08.267609 14321 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd29acbf656c48cd8d9cfd1fa99bcf38 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:50:08.270123 14328 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd29acbf656c48cd8d9cfd1fa99bcf38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd29acbf656c48cd8d9cfd1fa99bcf38. Latest consensus state: current_term: 1 leader_uuid: "dd29acbf656c48cd8d9cfd1fa99bcf38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd29acbf656c48cd8d9cfd1fa99bcf38" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 34401 } } }
I20260928 16:50:08.271019 14328 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd29acbf656c48cd8d9cfd1fa99bcf38 [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:08.270221 14327 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd29acbf656c48cd8d9cfd1fa99bcf38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd29acbf656c48cd8d9cfd1fa99bcf38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd29acbf656c48cd8d9cfd1fa99bcf38" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 34401 } } }
I20260928 16:50:08.273092 14327 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd29acbf656c48cd8d9cfd1fa99bcf38 [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:08.280237 14335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:50:08.286423 14335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:50:08.296983 14335 catalog_manager.cc:1347] Generated new cluster ID: 22c48f43a0b04c37a5c92b75b129a26c
I20260928 16:50:08.297125 14335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:50:08.322081 14335 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:50:08.322610 14335 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:50:08.330565 14335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd29acbf656c48cd8d9cfd1fa99bcf38: Generated new TSK 0
I20260928 16:50:08.331665 14335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:50:08.345762 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:34401
--builtin_ntp_servers=127.9.93.20:43709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 16:50:08.717481 14345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:08.717900 14345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:08.718039 14345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:08.726271 14345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:08.726416 14345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:08.726581 14345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:50:08.737346 14345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:43709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:34401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:08.738864 14345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:08.740986 14345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:08.756551 14353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:08.756624 14351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:08.756577 14350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:08.757432 14345 server_base.cc:1034] running on GCE node
I20260928 16:50:08.758123 14345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:08.759236 14345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:08.760567 14345 hybrid_clock.cc:648] HybridClock initialized: now 1790614208760484 us; error 62 us; skew 500 ppm
I20260928 16:50:08.760990 14345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:50:08.764002 14345 webserver.cc:460] Webserver started at http://127.9.93.1:34187/ using document root <none> and password file <none>
I20260928 16:50:08.764840 14345 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:08.764951 14345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:08.765257 14345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:08.768870 14345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "65fbf5b29643466bb8834675a140a425"
format_stamp: "Formatted at 2026-09-28 16:50:08 on dist-test-slave-wzcb"
server_key: "162194cf7e1b286ae1d0f45c6988d7f0"
server_key_iv: "868ffbca99029e0ea61e37603ff69f9f"
server_key_version: "encryptionkey@0"
I20260928 16:50:08.769774 14345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "65fbf5b29643466bb8834675a140a425"
format_stamp: "Formatted at 2026-09-28 16:50:08 on dist-test-slave-wzcb"
server_key: "162194cf7e1b286ae1d0f45c6988d7f0"
server_key_iv: "868ffbca99029e0ea61e37603ff69f9f"
server_key_version: "encryptionkey@0"
I20260928 16:50:08.776569 14345 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 16:50:08.781328 14359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:08.783133 14345 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260928 16:50:08.783377 14345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "65fbf5b29643466bb8834675a140a425"
format_stamp: "Formatted at 2026-09-28 16:50:08 on dist-test-slave-wzcb"
server_key: "162194cf7e1b286ae1d0f45c6988d7f0"
server_key_iv: "868ffbca99029e0ea61e37603ff69f9f"
server_key_version: "encryptionkey@0"
I20260928 16:50:08.783646 14345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:08.790973 14345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:08.792080 14345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:08.792378 14345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:08.793490 14345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:50:08.795287 14345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:08.795450 14345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260928 16:50:08.795554 14345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:08.795614 14345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:08.831684 14345 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:34143
I20260928 16:50:08.831756 14473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:34143 every 8 connection(s)
I20260928 16:50:08.833312 14345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:50:08.839530 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 14345
I20260928 16:50:08.839740 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:50:08.840433 9588 external_mini_cluster.cc:1428] Setting key 3c0bbee554310240cbfade7643a2fdda
I20260928 16:50:08.843428 14476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:08.836412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:43091 (local address 127.9.93.62:34401)
0928 16:50:08.836907 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:08.836911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:08.838422 (+ 1511us) server_negotiation.cc:407] Connection header received
0928 16:50:08.840572 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:08.840578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:08.840672 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:08.840784 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:08.842545 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:08.842552 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:08.842556 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:50:08.842579 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:08.842639 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:50:08.842657 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:08.842788 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:08.843042 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 16:50:08.843098 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":260,"threads_started":1}
I20260928 16:50:08.843964 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:34401
--builtin_ntp_servers=127.9.93.20:43709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 16:50:08.845276 14477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:08.837031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:34401 (local address 127.9.93.1:43091)
0928 16:50:08.838223 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:08.838270 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:08.840206 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:08.841148 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:08.841159 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:08.841310 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:08.842140 (+ 830us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:50:08.842143 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:08.842192 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:50:08.842340 (+ 148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:08.842783 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:08.842796 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:08.842804 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:50:08.843146 (+ 342us) client_negotiation.cc:240] Negotiation successful
0928 16:50:08.843704 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":338,"threads_started":1}
I20260928 16:50:08.847254 14474 heartbeater.cc:347] Connected to a master server at 127.9.93.62:34401
I20260928 16:50:08.847707 14474 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:08.848587 14474 heartbeater.cc:511] Master 127.9.93.62:34401 requested a full tablet report, sending...
I20260928 16:50:08.851372 14286 ts_manager.cc:194] Registered new tserver with Master: 65fbf5b29643466bb8834675a140a425 (127.9.93.1:34143)
W20260928 16:50:09.231366 14478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:09.231808 14478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:09.231982 14478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:09.244635 14478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:09.244797 14478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:09.245015 14478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:50:09.256063 14478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:43709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:34401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:09.257555 14478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:09.259687 14478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:09.275338 14484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:09.275174 14486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:09.275190 14483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:09.275934 14478 server_base.cc:1034] running on GCE node
I20260928 16:50:09.276769 14478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:09.277979 14478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:09.279214 14478 hybrid_clock.cc:648] HybridClock initialized: now 1790614209279131 us; error 83 us; skew 500 ppm
I20260928 16:50:09.279681 14478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:50:09.282465 14478 webserver.cc:460] Webserver started at http://127.9.93.2:46539/ using document root <none> and password file <none>
I20260928 16:50:09.283360 14478 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:09.283501 14478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:09.283864 14478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:09.287547 14478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "5661afdfa7a24f989ee570c646c50c08"
format_stamp: "Formatted at 2026-09-28 16:50:09 on dist-test-slave-wzcb"
server_key: "d2f1074cc5be9e285ebe88d29537714d"
server_key_iv: "444862d7f2f40a2f410939ea197f3d65"
server_key_version: "encryptionkey@0"
I20260928 16:50:09.288619 14478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "5661afdfa7a24f989ee570c646c50c08"
format_stamp: "Formatted at 2026-09-28 16:50:09 on dist-test-slave-wzcb"
server_key: "d2f1074cc5be9e285ebe88d29537714d"
server_key_iv: "444862d7f2f40a2f410939ea197f3d65"
server_key_version: "encryptionkey@0"
I20260928 16:50:09.295389 14478 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260928 16:50:09.300007 14492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:09.301868 14478 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260928 16:50:09.302124 14478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "5661afdfa7a24f989ee570c646c50c08"
format_stamp: "Formatted at 2026-09-28 16:50:09 on dist-test-slave-wzcb"
server_key: "d2f1074cc5be9e285ebe88d29537714d"
server_key_iv: "444862d7f2f40a2f410939ea197f3d65"
server_key_version: "encryptionkey@0"
I20260928 16:50:09.302434 14478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:09.309849 14478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:09.310957 14478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:09.311296 14478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:09.312614 14478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:50:09.314500 14478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:09.314626 14478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:09.314769 14478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:09.314854 14478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:09.352643 14478 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:37979
I20260928 16:50:09.352655 14605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:37979 every 8 connection(s)
I20260928 16:50:09.354295 14478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:50:09.360785 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 14478
I20260928 16:50:09.361007 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:50:09.361639 9588 external_mini_cluster.cc:1428] Setting key f8db2d66ef94b4027494a2f8bf1d5b67
I20260928 16:50:09.365582 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:34401
--builtin_ntp_servers=127.9.93.20:43709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 16:50:09.366690 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:09.357699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:37187 (local address 127.9.93.62:34401)
0928 16:50:09.358380 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:09.358388 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:09.360734 (+ 2346us) server_negotiation.cc:407] Connection header received
0928 16:50:09.363694 (+ 2960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:09.363700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:09.363847 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:09.363979 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:09.365679 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:09.365686 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:09.365691 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:50:09.365716 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:09.365781 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:50:09.365800 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:09.365931 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:09.366255 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 16:50:09.366342 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":352,"threads_started":1}
I20260928 16:50:09.368248 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:09.358601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:34401 (local address 127.9.93.2:37187)
0928 16:50:09.360459 (+ 1858us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:09.360533 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:09.363211 (+ 2678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:09.364358 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:09.364370 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:09.364502 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:09.365287 (+ 785us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:50:09.365291 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:09.365344 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:50:09.365463 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:09.365997 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:09.366009 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:09.366017 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:50:09.366345 (+ 328us) client_negotiation.cc:240] Negotiation successful
0928 16:50:09.366775 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"thread_start_us":748,"threads_started":1}
I20260928 16:50:09.370100 14606 heartbeater.cc:347] Connected to a master server at 127.9.93.62:34401
I20260928 16:50:09.370478 14606 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:09.371234 14606 heartbeater.cc:511] Master 127.9.93.62:34401 requested a full tablet report, sending...
I20260928 16:50:09.373090 14286 ts_manager.cc:194] Registered new tserver with Master: 5661afdfa7a24f989ee570c646c50c08 (127.9.93.2:37979)
W20260928 16:50:09.741876 14610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:09.742236 14610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:09.742342 14610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:09.750607 14610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:09.750744 14610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:09.750872 14610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:50:09.761138 14610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:43709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:34401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:09.762640 14610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:09.764740 14610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:09.780272 14616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:09.780232 14618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:09.780304 14615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:09.781297 14610 server_base.cc:1034] running on GCE node
I20260928 16:50:09.782088 14610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:09.783257 14610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:09.784600 14610 hybrid_clock.cc:648] HybridClock initialized: now 1790614209784497 us; error 51 us; skew 500 ppm
I20260928 16:50:09.784991 14610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:50:09.787742 14610 webserver.cc:460] Webserver started at http://127.9.93.3:40373/ using document root <none> and password file <none>
I20260928 16:50:09.788645 14610 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:09.788777 14610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:09.789135 14610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:09.792948 14610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "3e4f0cf02eaa41be84a227cf12f1a0ae"
format_stamp: "Formatted at 2026-09-28 16:50:09 on dist-test-slave-wzcb"
server_key: "0a4f36d953a3e6830a10bc69fca5c021"
server_key_iv: "6353b84beebcab8dd745dc5f8e0a4952"
server_key_version: "encryptionkey@0"
I20260928 16:50:09.793921 14610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "3e4f0cf02eaa41be84a227cf12f1a0ae"
format_stamp: "Formatted at 2026-09-28 16:50:09 on dist-test-slave-wzcb"
server_key: "0a4f36d953a3e6830a10bc69fca5c021"
server_key_iv: "6353b84beebcab8dd745dc5f8e0a4952"
server_key_version: "encryptionkey@0"
I20260928 16:50:09.800691 14610 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 16:50:09.805207 14624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:09.807031 14610 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:50:09.807261 14610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "3e4f0cf02eaa41be84a227cf12f1a0ae"
format_stamp: "Formatted at 2026-09-28 16:50:09 on dist-test-slave-wzcb"
server_key: "0a4f36d953a3e6830a10bc69fca5c021"
server_key_iv: "6353b84beebcab8dd745dc5f8e0a4952"
server_key_version: "encryptionkey@0"
I20260928 16:50:09.807654 14610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:09.814560 14610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:09.815676 14610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:09.815992 14610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:09.817390 14610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:50:09.819182 14610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:09.819320 14610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:09.819453 14610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:09.819530 14610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:09.855156 14474 heartbeater.cc:503] Master 127.9.93.62:34401 was elected leader, sending a full tablet report...
I20260928 16:50:09.855801 14610 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:44199
I20260928 16:50:09.855827 14738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:44199 every 8 connection(s)
I20260928 16:50:09.857563 14610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:50:09.860742 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 14610
I20260928 16:50:09.860936 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:50:09.861538 9588 external_mini_cluster.cc:1428] Setting key 20651cf37989cca9203a9643d68fea0b
I20260928 16:50:09.868149 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:09.860502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:59623 (local address 127.9.93.62:34401)
0928 16:50:09.860677 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:09.860681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:09.863070 (+ 2389us) server_negotiation.cc:407] Connection header received
0928 16:50:09.865593 (+ 2523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:09.865598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:09.865662 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:09.865744 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:09.867419 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:09.867424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:09.867428 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:50:09.867448 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:09.867486 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:50:09.867498 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:09.867610 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:09.867855 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 16:50:09.867919 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:50:09.869529 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:09.861424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:34401 (local address 127.9.93.3:59623)
0928 16:50:09.862872 (+ 1448us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:09.862932 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:09.865280 (+ 2348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:09.866106 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:09.866118 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:09.866248 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:09.866985 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:50:09.866987 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:09.867039 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:50:09.867173 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:09.867608 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:09.867617 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:09.867625 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:50:09.867934 (+ 309us) client_negotiation.cc:240] Negotiation successful
0928 16:50:09.868251 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":445,"threads_started":1}
I20260928 16:50:09.871495 14739 heartbeater.cc:347] Connected to a master server at 127.9.93.62:34401
I20260928 16:50:09.871892 14739 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:09.872689 14739 heartbeater.cc:511] Master 127.9.93.62:34401 requested a full tablet report, sending...
I20260928 16:50:09.874554 14286 ts_manager.cc:194] Registered new tserver with Master: 3e4f0cf02eaa41be84a227cf12f1a0ae (127.9.93.3:44199)
I20260928 16:50:09.878198 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:50:09.886106 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:09.883243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45646 (local address 127.9.93.62:34401)
0928 16:50:09.883483 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:09.883488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:09.884033 (+ 545us) server_negotiation.cc:407] Connection header received
0928 16:50:09.884239 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:09.884246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:09.884314 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:09.884403 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:09.885407 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:09.885413 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:09.885418 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:50:09.885434 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:09.885470 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:50:09.885482 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:09.885616 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:09.885710 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 16:50:09.885784 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260928 16:50:09.891091 14286 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:50:09.898594 14286 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:50:09.919406 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:09.914688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48314 (local address 127.9.93.2:37979)
0928 16:50:09.915374 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:09.915383 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:09.915420 (+ 37us) server_negotiation.cc:407] Connection header received
0928 16:50:09.915518 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:09.915526 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:09.915874 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:09.916067 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:09.918375 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:09.918386 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:09.918406 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:50:09.918426 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:09.918509 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:50:09.918540 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:09.918753 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:09.918898 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 16:50:09.919042 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"spinlock_wait_cycles":27008,"thread_start_us":387,"threads_started":1}
I20260928 16:50:09.919518 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:09.913827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:37979 (local address 127.0.0.1:48314)
0928 16:50:09.914723 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:09.914773 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:09.914912 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:09.916134 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:09.916143 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:09.916259 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:09.917215 (+ 956us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:50:09.917218 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:09.917257 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:50:09.917371 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:09.918696 (+ 1325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:09.918710 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:09.918718 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:50:09.918913 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 16:50:09.919039 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":382,"threads_started":1}
I20260928 16:50:09.922864 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:09.916245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:44199 (local address 127.0.0.1:40524)
0928 16:50:09.918475 (+ 2230us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:09.918491 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:09.918612 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:09.919935 (+ 1323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:09.919940 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:09.919981 (+ 41us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:09.920785 (+ 804us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:50:09.920788 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:09.920808 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:50:09.920860 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:09.922304 (+ 1444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:09.922308 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:09.922313 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 16:50:09.922437 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 16:50:09.922502 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2109,"thread_start_us":1970,"threads_started":1}
I20260928 16:50:09.922986 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:09.915379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37930 (local address 127.9.93.1:34143)
0928 16:50:09.919225 (+ 3846us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:09.919235 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:09.919269 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:50:09.919383 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:09.919394 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:09.919709 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:09.919890 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:09.921889 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:09.921899 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:09.921915 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:50:09.921936 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:09.922011 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:50:09.922043 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:09.922227 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:09.922453 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 16:50:09.922588 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3652,"thread_start_us":3474,"threads_started":1}
I20260928 16:50:09.923827 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:09.914161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:34143 (local address 127.0.0.1:37930)
0928 16:50:09.915399 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:09.915422 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:09.915533 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:09.920931 (+ 5398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:09.920935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:09.920956 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:09.921620 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:50:09.921624 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:09.921643 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:50:09.921693 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:09.922289 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:09.922294 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:09.922300 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:50:09.923266 (+ 966us) client_negotiation.cc:240] Negotiation successful
0928 16:50:09.923359 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058,"thread_start_us":1169,"threads_started":1}
I20260928 16:50:09.923943 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:09.916926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40524 (local address 127.9.93.3:44199)
0928 16:50:09.918039 (+ 1113us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:09.918050 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:09.918607 (+ 557us) server_negotiation.cc:407] Connection header received
0928 16:50:09.919475 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:09.919485 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:09.919727 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:09.920482 (+ 755us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:09.922051 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:09.922059 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:09.922072 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:50:09.922092 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:09.922157 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:50:09.922180 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:09.923446 (+ 1266us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:09.923510 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 16:50:09.923606 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":914,"thread_start_us":727,"threads_started":1}
I20260928 16:50:09.925032 14540 tablet_service.cc:1467] Processing CreateTablet for tablet fb9afcddfe0c4e348e252a723cedbf51 (DEFAULT_TABLE table=test-table [id=46bfa1ac53ac413195f8e1254f06ef72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:50:09.926882 14540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9afcddfe0c4e348e252a723cedbf51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:09.928251 14408 tablet_service.cc:1467] Processing CreateTablet for tablet fb9afcddfe0c4e348e252a723cedbf51 (DEFAULT_TABLE table=test-table [id=46bfa1ac53ac413195f8e1254f06ef72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:50:09.928560 14673 tablet_service.cc:1467] Processing CreateTablet for tablet fb9afcddfe0c4e348e252a723cedbf51 (DEFAULT_TABLE table=test-table [id=46bfa1ac53ac413195f8e1254f06ef72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:50:09.929867 14673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9afcddfe0c4e348e252a723cedbf51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:09.929908 14408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9afcddfe0c4e348e252a723cedbf51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:09.943696 14757 tablet_bootstrap.cc:492] T fb9afcddfe0c4e348e252a723cedbf51 P 3e4f0cf02eaa41be84a227cf12f1a0ae: Bootstrap starting.
I20260928 16:50:09.946496 14758 tablet_bootstrap.cc:492] T fb9afcddfe0c4e348e252a723cedbf51 P 5661afdfa7a24f989ee570c646c50c08: Bootstrap starting.
I20260928 16:50:09.950623 14758 tablet_bootstrap.cc:654] T fb9afcddfe0c4e348e252a723cedbf51 P 5661afdfa7a24f989ee570c646c50c08: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:09.950625 14757 tablet_bootstrap.cc:654] T fb9afcddfe0c4e348e252a723cedbf51 P 3e4f0cf02eaa41be84a227cf12f1a0ae: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:09.951817 14759 tablet_bootstrap.cc:492] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425: Bootstrap starting.
I20260928 16:50:09.952342 14758 log.cc:824] T fb9afcddfe0c4e348e252a723cedbf51 P 5661afdfa7a24f989ee570c646c50c08: Log is configured to *not* fsync() on all Append() calls
I20260928 16:50:09.953222 14757 log.cc:824] T fb9afcddfe0c4e348e252a723cedbf51 P 3e4f0cf02eaa41be84a227cf12f1a0ae: Log is configured to *not* fsync() on all Append() calls
I20260928 16:50:09.956059 14759 tablet_bootstrap.cc:654] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:09.956223 14758 tablet_bootstrap.cc:492] T fb9afcddfe0c4e348e252a723cedbf51 P 5661afdfa7a24f989ee570c646c50c08: No bootstrap required, opened a new log
I20260928 16:50:09.956589 14758 ts_tablet_manager.cc:1397] T fb9afcddfe0c4e348e252a723cedbf51 P 5661afdfa7a24f989ee570c646c50c08: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20260928 16:50:09.957477 14759 log.cc:824] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425: Log is configured to *not* fsync() on all Append() calls
I20260928 16:50:09.959183 14757 tablet_bootstrap.cc:492] T fb9afcddfe0c4e348e252a723cedbf51 P 3e4f0cf02eaa41be84a227cf12f1a0ae: No bootstrap required, opened a new log
I20260928 16:50:09.959574 14757 ts_tablet_manager.cc:1397] T fb9afcddfe0c4e348e252a723cedbf51 P 3e4f0cf02eaa41be84a227cf12f1a0ae: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.000s
I20260928 16:50:09.960584 14759 tablet_bootstrap.cc:492] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425: No bootstrap required, opened a new log
I20260928 16:50:09.960856 14759 ts_tablet_manager.cc:1397] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 16:50:09.968091 14758 raft_consensus.cc:348] T fb9afcddfe0c4e348e252a723cedbf51 P 5661afdfa7a24f989ee570c646c50c08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5661afdfa7a24f989ee570c646c50c08" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 37979 } } peers { permanent_uuid: "65fbf5b29643466bb8834675a140a425" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 34143 } } peers { permanent_uuid: "3e4f0cf02eaa41be84a227cf12f1a0ae" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44199 } }
I20260928 16:50:09.968715 14758 raft_consensus.cc:374] T fb9afcddfe0c4e348e252a723cedbf51 P 5661afdfa7a24f989ee570c646c50c08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:09.968827 14758 raft_consensus.cc:729] T fb9afcddfe0c4e348e252a723cedbf51 P 5661afdfa7a24f989ee570c646c50c08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5661afdfa7a24f989ee570c646c50c08, State: Initialized, Role: FOLLOWER
I20260928 16:50:09.969414 14759 raft_consensus.cc:348] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5661afdfa7a24f989ee570c646c50c08" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 37979 } } peers { permanent_uuid: "65fbf5b29643466bb8834675a140a425" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 34143 } } peers { permanent_uuid: "3e4f0cf02eaa41be84a227cf12f1a0ae" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44199 } }
I20260928 16:50:09.969810 14759 raft_consensus.cc:374] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:09.969640 14758 consensus_queue.cc:260] T fb9afcddfe0c4e348e252a723cedbf51 P 5661afdfa7a24f989ee570c646c50c08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5661afdfa7a24f989ee570c646c50c08" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 37979 } } peers { permanent_uuid: "65fbf5b29643466bb8834675a140a425" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 34143 } } peers { permanent_uuid: "3e4f0cf02eaa41be84a227cf12f1a0ae" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44199 } }
I20260928 16:50:09.969913 14759 raft_consensus.cc:729] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65fbf5b29643466bb8834675a140a425, State: Initialized, Role: FOLLOWER
I20260928 16:50:09.970528 14759 consensus_queue.cc:260] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5661afdfa7a24f989ee570c646c50c08" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 37979 } } peers { permanent_uuid: "65fbf5b29643466bb8834675a140a425" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 34143 } } peers { permanent_uuid: "3e4f0cf02eaa41be84a227cf12f1a0ae" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44199 } }
I20260928 16:50:09.971983 14757 raft_consensus.cc:348] T fb9afcddfe0c4e348e252a723cedbf51 P 3e4f0cf02eaa41be84a227cf12f1a0ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5661afdfa7a24f989ee570c646c50c08" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 37979 } } peers { permanent_uuid: "65fbf5b29643466bb8834675a140a425" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 34143 } } peers { permanent_uuid: "3e4f0cf02eaa41be84a227cf12f1a0ae" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44199 } }
I20260928 16:50:09.972638 14757 raft_consensus.cc:374] T fb9afcddfe0c4e348e252a723cedbf51 P 3e4f0cf02eaa41be84a227cf12f1a0ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:09.972692 14759 ts_tablet_manager.cc:1428] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260928 16:50:09.972756 14757 raft_consensus.cc:729] T fb9afcddfe0c4e348e252a723cedbf51 P 3e4f0cf02eaa41be84a227cf12f1a0ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e4f0cf02eaa41be84a227cf12f1a0ae, State: Initialized, Role: FOLLOWER
I20260928 16:50:09.973632 14757 consensus_queue.cc:260] T fb9afcddfe0c4e348e252a723cedbf51 P 3e4f0cf02eaa41be84a227cf12f1a0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5661afdfa7a24f989ee570c646c50c08" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 37979 } } peers { permanent_uuid: "65fbf5b29643466bb8834675a140a425" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 34143 } } peers { permanent_uuid: "3e4f0cf02eaa41be84a227cf12f1a0ae" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44199 } }
I20260928 16:50:09.975255 14606 heartbeater.cc:503] Master 127.9.93.62:34401 was elected leader, sending a full tablet report...
I20260928 16:50:09.976347 14758 ts_tablet_manager.cc:1428] T fb9afcddfe0c4e348e252a723cedbf51 P 5661afdfa7a24f989ee570c646c50c08: Time spent starting tablet: real 0.020s user 0.015s sys 0.004s
I20260928 16:50:09.977806 14739 heartbeater.cc:503] Master 127.9.93.62:34401 was elected leader, sending a full tablet report...
I20260928 16:50:09.978681 14757 ts_tablet_manager.cc:1428] T fb9afcddfe0c4e348e252a723cedbf51 P 3e4f0cf02eaa41be84a227cf12f1a0ae: Time spent starting tablet: real 0.019s user 0.017s sys 0.000s
I20260928 16:50:10.032593 14764 raft_consensus.cc:482] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:50:10.032956 14764 raft_consensus.cc:504] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5661afdfa7a24f989ee570c646c50c08" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 37979 } } peers { permanent_uuid: "65fbf5b29643466bb8834675a140a425" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 34143 } } peers { permanent_uuid: "3e4f0cf02eaa41be84a227cf12f1a0ae" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44199 } }
I20260928 16:50:10.035171 14764 leader_election.cc:288] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5661afdfa7a24f989ee570c646c50c08 (127.9.93.2:37979), 3e4f0cf02eaa41be84a227cf12f1a0ae (127.9.93.3:44199)
I20260928 16:50:10.039481 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:10.035760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:43203 (local address 127.9.93.3:44199)
0928 16:50:10.035994 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:10.036000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:10.037276 (+ 1276us) server_negotiation.cc:407] Connection header received
0928 16:50:10.037360 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:10.037365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:10.037432 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:10.037559 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:10.038757 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:10.038764 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:10.038769 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:50:10.038792 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:10.038838 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:50:10.038855 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:10.039005 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:10.039067 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 16:50:10.039141 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 16:50:10.039505 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:10.035827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:58447 (local address 127.9.93.2:37979)
0928 16:50:10.036020 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:10.036026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:10.037168 (+ 1142us) server_negotiation.cc:407] Connection header received
0928 16:50:10.037342 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:10.037348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:10.037432 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:10.037558 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:10.038757 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:10.038764 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:10.038770 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:50:10.038792 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:10.038838 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:50:10.038855 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:10.038983 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:10.039067 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 16:50:10.039145 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 16:50:10.039745 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:10.035924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:37979 (local address 127.9.93.1:58447)
0928 16:50:10.036990 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:10.037016 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:10.037192 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:10.037584 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:10.037588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:10.037633 (+ 45us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:10.038474 (+ 841us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:50:10.038478 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:10.038503 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:50:10.038564 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:10.038963 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:10.038968 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:10.038973 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 16:50:10.039139 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 16:50:10.039205 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"thread_start_us":491,"threads_started":1}
I20260928 16:50:10.040166 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:10.035943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:44199 (local address 127.9.93.1:43203)
0928 16:50:10.037142 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:10.037158 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:10.037260 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:10.037582 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:10.037587 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:10.037615 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:10.038527 (+ 912us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:50:10.038530 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:10.038542 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:50:10.038570 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:10.038963 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:10.038968 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:10.038973 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 16:50:10.039136 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 16:50:10.039215 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1080,"mutex_wait_us":50,"spinlock_wait_cycles":29440,"thread_start_us":832,"threads_started":1}
I20260928 16:50:10.041079 14560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9afcddfe0c4e348e252a723cedbf51" candidate_uuid: "65fbf5b29643466bb8834675a140a425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5661afdfa7a24f989ee570c646c50c08" is_pre_election: true
I20260928 16:50:10.041296 14693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9afcddfe0c4e348e252a723cedbf51" candidate_uuid: "65fbf5b29643466bb8834675a140a425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e4f0cf02eaa41be84a227cf12f1a0ae" is_pre_election: true
I20260928 16:50:10.041739 14560 raft_consensus.cc:2454] T fb9afcddfe0c4e348e252a723cedbf51 P 5661afdfa7a24f989ee570c646c50c08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65fbf5b29643466bb8834675a140a425 in term 0.
I20260928 16:50:10.041960 14693 raft_consensus.cc:2454] T fb9afcddfe0c4e348e252a723cedbf51 P 3e4f0cf02eaa41be84a227cf12f1a0ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65fbf5b29643466bb8834675a140a425 in term 0.
I20260928 16:50:10.042552 14362 leader_election.cc:302] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5661afdfa7a24f989ee570c646c50c08, 65fbf5b29643466bb8834675a140a425; no voters:
I20260928 16:50:10.043005 14764 raft_consensus.cc:2781] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:50:10.043206 14764 raft_consensus.cc:482] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:50:10.043365 14764 raft_consensus.cc:3037] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:10.045866 14764 raft_consensus.cc:504] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5661afdfa7a24f989ee570c646c50c08" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 37979 } } peers { permanent_uuid: "65fbf5b29643466bb8834675a140a425" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 34143 } } peers { permanent_uuid: "3e4f0cf02eaa41be84a227cf12f1a0ae" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44199 } }
I20260928 16:50:10.046582 14764 leader_election.cc:288] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425 [CANDIDATE]: Term 1 election: Requested vote from peers 5661afdfa7a24f989ee570c646c50c08 (127.9.93.2:37979), 3e4f0cf02eaa41be84a227cf12f1a0ae (127.9.93.3:44199)
I20260928 16:50:10.047328 14693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9afcddfe0c4e348e252a723cedbf51" candidate_uuid: "65fbf5b29643466bb8834675a140a425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e4f0cf02eaa41be84a227cf12f1a0ae"
I20260928 16:50:10.047648 14693 raft_consensus.cc:3037] T fb9afcddfe0c4e348e252a723cedbf51 P 3e4f0cf02eaa41be84a227cf12f1a0ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:10.047624 14560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9afcddfe0c4e348e252a723cedbf51" candidate_uuid: "65fbf5b29643466bb8834675a140a425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5661afdfa7a24f989ee570c646c50c08"
I20260928 16:50:10.047863 14560 raft_consensus.cc:3037] T fb9afcddfe0c4e348e252a723cedbf51 P 5661afdfa7a24f989ee570c646c50c08 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:10.050417 14693 raft_consensus.cc:2454] T fb9afcddfe0c4e348e252a723cedbf51 P 3e4f0cf02eaa41be84a227cf12f1a0ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65fbf5b29643466bb8834675a140a425 in term 1.
I20260928 16:50:10.050431 14560 raft_consensus.cc:2454] T fb9afcddfe0c4e348e252a723cedbf51 P 5661afdfa7a24f989ee570c646c50c08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65fbf5b29643466bb8834675a140a425 in term 1.
I20260928 16:50:10.051011 14360 leader_election.cc:302] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e4f0cf02eaa41be84a227cf12f1a0ae, 65fbf5b29643466bb8834675a140a425; no voters:
I20260928 16:50:10.051414 14764 raft_consensus.cc:2781] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:50:10.051885 14764 raft_consensus.cc:686] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425 [term 1 LEADER]: Becoming Leader. State: Replica: 65fbf5b29643466bb8834675a140a425, State: Running, Role: LEADER
I20260928 16:50:10.052409 14764 consensus_queue.cc:237] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5661afdfa7a24f989ee570c646c50c08" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 37979 } } peers { permanent_uuid: "65fbf5b29643466bb8834675a140a425" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 34143 } } peers { permanent_uuid: "3e4f0cf02eaa41be84a227cf12f1a0ae" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44199 } }
I20260928 16:50:10.057679 14286 catalog_manager.cc:5495] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65fbf5b29643466bb8834675a140a425 (127.9.93.1). New cstate: current_term: 1 leader_uuid: "65fbf5b29643466bb8834675a140a425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5661afdfa7a24f989ee570c646c50c08" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 37979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65fbf5b29643466bb8834675a140a425" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 34143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e4f0cf02eaa41be84a227cf12f1a0ae" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44199 } health_report { overall_health: UNKNOWN } } }
I20260928 16:50:10.083346 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:10.081222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37940 (local address 127.9.93.1:34143)
0928 16:50:10.081489 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:10.081495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:10.081518 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:50:10.081627 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:10.081631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:10.081727 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:10.081824 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:10.082711 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:10.082716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:10.082719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:50:10.082739 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:10.082778 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:50:10.082791 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:10.082909 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:10.082947 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 16:50:10.083005 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260928 16:50:10.086140 14475 tablet.cc:2369] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:50:10.094121 14693 raft_consensus.cc:1261] T fb9afcddfe0c4e348e252a723cedbf51 P 3e4f0cf02eaa41be84a227cf12f1a0ae [term 1 FOLLOWER]: Refusing update from remote peer 65fbf5b29643466bb8834675a140a425: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:50:10.094117 14560 raft_consensus.cc:1261] T fb9afcddfe0c4e348e252a723cedbf51 P 5661afdfa7a24f989ee570c646c50c08 [term 1 FOLLOWER]: Refusing update from remote peer 65fbf5b29643466bb8834675a140a425: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:50:10.095477 14768 consensus_queue.cc:1035] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5661afdfa7a24f989ee570c646c50c08" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 37979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:50:10.095943 14764 consensus_queue.cc:1035] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e4f0cf02eaa41be84a227cf12f1a0ae" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 16:50:10.106657 14607 tablet.cc:2369] T fb9afcddfe0c4e348e252a723cedbf51 P 5661afdfa7a24f989ee570c646c50c08: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:50:10.109368 14740 tablet.cc:2369] T fb9afcddfe0c4e348e252a723cedbf51 P 3e4f0cf02eaa41be84a227cf12f1a0ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:50:10.112955 14771 mvcc.cc:205] Tried to move back new op lower bound from 7334355804531204096 to 7334355804378763264. Current Snapshot: MvccSnapshot[applied={T|T < 7334355804531204096}]
I20260928 16:50:10.134521 14286 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:50:10.134820 14286 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:50:10.139364 14286 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd29acbf656c48cd8d9cfd1fa99bcf38: Sending DeleteTablet for 3 replicas of tablet fb9afcddfe0c4e348e252a723cedbf51
I20260928 16:50:10.140892 14540 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9afcddfe0c4e348e252a723cedbf51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:10 UTC) from {username='slave'} at 127.0.0.1:48314
I20260928 16:50:10.141258 14408 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9afcddfe0c4e348e252a723cedbf51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:10 UTC) from {username='slave'} at 127.0.0.1:37930
I20260928 16:50:10.142198 14673 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9afcddfe0c4e348e252a723cedbf51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:10 UTC) from {username='slave'} at 127.0.0.1:40524
I20260928 16:50:10.142427 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 14345
I20260928 16:50:10.145437 14782 tablet_replica.cc:330] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425: stopping tablet replica
I20260928 16:50:10.146006 14782 raft_consensus.cc:2229] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:50:10.146860 14782 raft_consensus.cc:2258] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:10.149755 14783 tablet_replica.cc:330] T fb9afcddfe0c4e348e252a723cedbf51 P 3e4f0cf02eaa41be84a227cf12f1a0ae: stopping tablet replica
I20260928 16:50:10.150329 14783 raft_consensus.cc:2229] T fb9afcddfe0c4e348e252a723cedbf51 P 3e4f0cf02eaa41be84a227cf12f1a0ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:50:10.150456 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:10.144078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37954 (local address 127.9.93.1:34143)
0928 16:50:10.144281 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:10.144289 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:10.147798 (+ 3509us) server_negotiation.cc:407] Connection header received
0928 16:50:10.148049 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:10.148055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:10.148149 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:10.148272 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:10.149464 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:10.149471 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:10.149477 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:50:10.149504 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:10.149557 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:50:10.149576 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:10.149755 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:10.149993 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 16:50:10.150096 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 16:50:10.150980 14783 raft_consensus.cc:2258] T fb9afcddfe0c4e348e252a723cedbf51 P 3e4f0cf02eaa41be84a227cf12f1a0ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:10.151706 14469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:10.152515 14781 tablet_replica.cc:330] T fb9afcddfe0c4e348e252a723cedbf51 P 5661afdfa7a24f989ee570c646c50c08: stopping tablet replica
I20260928 16:50:10.153038 14781 raft_consensus.cc:2229] T fb9afcddfe0c4e348e252a723cedbf51 P 5661afdfa7a24f989ee570c646c50c08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:50:10.153637 14781 raft_consensus.cc:2258] T fb9afcddfe0c4e348e252a723cedbf51 P 5661afdfa7a24f989ee570c646c50c08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:10.153657 14783 ts_tablet_manager.cc:1905] T fb9afcddfe0c4e348e252a723cedbf51 P 3e4f0cf02eaa41be84a227cf12f1a0ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:50:10.155812 14781 ts_tablet_manager.cc:1905] T fb9afcddfe0c4e348e252a723cedbf51 P 5661afdfa7a24f989ee570c646c50c08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:50:10.163494 14781 ts_tablet_manager.cc:1918] T fb9afcddfe0c4e348e252a723cedbf51 P 5661afdfa7a24f989ee570c646c50c08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:50:10.163494 14783 ts_tablet_manager.cc:1918] T fb9afcddfe0c4e348e252a723cedbf51 P 3e4f0cf02eaa41be84a227cf12f1a0ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:50:10.163735 14783 log.cc:1196] T fb9afcddfe0c4e348e252a723cedbf51 P 3e4f0cf02eaa41be84a227cf12f1a0ae: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/fb9afcddfe0c4e348e252a723cedbf51
I20260928 16:50:10.163734 14781 log.cc:1196] T fb9afcddfe0c4e348e252a723cedbf51 P 5661afdfa7a24f989ee570c646c50c08: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/fb9afcddfe0c4e348e252a723cedbf51
I20260928 16:50:10.164362 14783 ts_tablet_manager.cc:1939] T fb9afcddfe0c4e348e252a723cedbf51 P 3e4f0cf02eaa41be84a227cf12f1a0ae: Deleting consensus metadata
I20260928 16:50:10.164362 14781 ts_tablet_manager.cc:1939] T fb9afcddfe0c4e348e252a723cedbf51 P 5661afdfa7a24f989ee570c646c50c08: Deleting consensus metadata
I20260928 16:50:10.167071 14270 catalog_manager.cc:4841] TS 3e4f0cf02eaa41be84a227cf12f1a0ae (127.9.93.3:44199): tablet fb9afcddfe0c4e348e252a723cedbf51 (table test-table [id=46bfa1ac53ac413195f8e1254f06ef72]) successfully deleted
I20260928 16:50:10.167448 14272 catalog_manager.cc:4841] TS 5661afdfa7a24f989ee570c646c50c08 (127.9.93.2:37979): tablet fb9afcddfe0c4e348e252a723cedbf51 (table test-table [id=46bfa1ac53ac413195f8e1254f06ef72]) successfully deleted
I20260928 16:50:10.153611 14782 ts_tablet_manager.cc:1905] T fb9afcddfe0c4e348e252a723cedbf51 P 65fbf5b29643466bb8834675a140a425: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:50:10.258949 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 14345
W20260928 16:50:10.276444 14273 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.93.1:34143 (error 108)
I20260928 16:50:10.276671 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 14478
I20260928 16:50:10.278652 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:10.278072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:34143 (local address 127.0.0.1:37966)
0928 16:50:10.278272 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:10.278424 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260928 16:50:10.278913 14273 catalog_manager.cc:4584] TS 65fbf5b29643466bb8834675a140a425 (127.9.93.1:34143): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb9afcddfe0c4e348e252a723cedbf51: Network error: Client connection negotiation failed: client connection to 127.9.93.1:34143: connect: Connection refused (error 111)
I20260928 16:50:10.279949 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:10.277596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48320 (local address 127.9.93.2:37979)
0928 16:50:10.277788 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:10.277792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:10.277813 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:50:10.278005 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:10.278012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:10.278131 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:10.278255 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:10.279325 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:10.279334 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:10.279338 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:50:10.279361 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:10.279399 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:50:10.279415 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:10.279528 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:10.279617 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 16:50:10.279675 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 16:50:10.280652 14601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:10.332919 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:10.332399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:34143 (local address 127.0.0.1:37978)
0928 16:50:10.332636 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:10.332741 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 16:50:10.334268 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:10.333762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:34143 (local address 127.0.0.1:37986)
0928 16:50:10.333992 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:10.334121 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260928 16:50:10.377267 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 14478
I20260928 16:50:10.395049 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 14610
I20260928 16:50:10.395102 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:10.394376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:34143 (local address 127.0.0.1:38000)
0928 16:50:10.394780 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:10.394884 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266}
I20260928 16:50:10.396271 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:10.395814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:34143 (local address 127.0.0.1:38002)
0928 16:50:10.396002 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:10.396111 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 16:50:10.398131 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:10.395975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40538 (local address 127.9.93.3:44199)
0928 16:50:10.396174 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:10.396181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:10.396227 (+ 46us) server_negotiation.cc:407] Connection header received
0928 16:50:10.396440 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:10.396447 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:10.396554 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:10.396644 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:10.397546 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:10.397551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:10.397554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:50:10.397570 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:10.397604 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:50:10.397616 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:10.397727 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:10.397770 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 16:50:10.397822 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 16:50:10.398841 14734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:10.477152 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:10.476594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:34143 (local address 127.0.0.1:38008)
0928 16:50:10.476861 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:10.476983 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260928 16:50:10.478394 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:10.477950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:34143 (local address 127.0.0.1:38024)
0928 16:50:10.478171 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:10.478252 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260928 16:50:10.493227 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 14610
I20260928 16:50:10.507949 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 14255
I20260928 16:50:10.509258 14316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:10.620487 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 14255
2026-09-28T16:50:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2923 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 16:50:10.662060 9588 mini_oidc.cc:152] Starting JWKS server
I20260928 16:50:10.667856 9588 webserver.cc:460] Webserver started at https://127.0.0.1:46783/ using document root <none> and password file <none>
I20260928 16:50:10.668275 9588 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 16:50:10.669031 9588 webserver.cc:460] Webserver started at http://127.0.0.1:40433/ using document root <none> and password file <none>
2026-09-28T16:50:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:50:10Z Disabled control of system clock
I20260928 16:50:10.701498 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:33507
--webserver_interface=127.9.93.62
--webserver_port=0
--builtin_ntp_servers=127.9.93.20:46633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:33507
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46783/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790614166338986-9588-0/testchainca.cert with env {}
W20260928 16:50:11.072753 14795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:11.073119 14795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:11.073242 14795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:11.081017 14795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:50:11.081176 14795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:11.081220 14795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:11.081301 14795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:50:11.081370 14795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 16:50:11.081458 14795 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 16:50:11.081516 14795 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46783/jwks.json
I20260928 16:50:11.092623 14795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:46633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:33507
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:33507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46783/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790614166338986-9588-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:11.094193 14795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:11.095806 14795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:11.109287 14800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:11.109205 14801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:11.109251 14803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:11.110481 14795 server_base.cc:1034] running on GCE node
I20260928 16:50:11.111546 14795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:11.113332 14795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:11.114569 14795 hybrid_clock.cc:648] HybridClock initialized: now 1790614211114497 us; error 68 us; skew 500 ppm
I20260928 16:50:11.115043 14795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:50:11.117834 14795 webserver.cc:460] Webserver started at http://127.9.93.62:35743/ using document root <none> and password file <none>
I20260928 16:50:11.118758 14795 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:11.118883 14795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:11.119248 14795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:11.123133 14795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "12fa6a7b1a9c4a728ef024bc09a101eb"
format_stamp: "Formatted at 2026-09-28 16:50:11 on dist-test-slave-wzcb"
server_key: "ac18998ee288b06953fd173f9b9da1dc"
server_key_iv: "934bb7d7bbe669d2ddfa134e739fa0ac"
server_key_version: "encryptionkey@0"
I20260928 16:50:11.124290 14795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "12fa6a7b1a9c4a728ef024bc09a101eb"
format_stamp: "Formatted at 2026-09-28 16:50:11 on dist-test-slave-wzcb"
server_key: "ac18998ee288b06953fd173f9b9da1dc"
server_key_iv: "934bb7d7bbe669d2ddfa134e739fa0ac"
server_key_version: "encryptionkey@0"
I20260928 16:50:11.130821 14795 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:50:11.135293 14809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:11.137133 14795 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:50:11.137378 14795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "12fa6a7b1a9c4a728ef024bc09a101eb"
format_stamp: "Formatted at 2026-09-28 16:50:11 on dist-test-slave-wzcb"
server_key: "ac18998ee288b06953fd173f9b9da1dc"
server_key_iv: "934bb7d7bbe669d2ddfa134e739fa0ac"
server_key_version: "encryptionkey@0"
I20260928 16:50:11.137683 14795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:11.178115 14795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:11.179105 14795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:11.179455 14795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:11.200678 14860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:33507 every 8 connection(s)
I20260928 16:50:11.200702 14795 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:33507
I20260928 16:50:11.203207 14795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:50:11.206529 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 14795
I20260928 16:50:11.206790 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:50:11.207521 9588 external_mini_cluster.cc:1428] Setting key 8632b3a4c8a29a4379d73d15b1b78bf6
I20260928 16:50:11.208884 14861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:11.226876 14861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12fa6a7b1a9c4a728ef024bc09a101eb: Bootstrap starting.
I20260928 16:50:11.227389 14864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:11.210130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50568 (local address 127.9.93.62:33507)
0928 16:50:11.211525 (+ 1395us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:11.211544 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:11.212025 (+ 481us) server_negotiation.cc:407] Connection header received
0928 16:50:11.214596 (+ 2571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:11.214638 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:11.215562 (+ 924us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:11.216223 (+ 661us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:11.216978 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:11.219241 (+ 2263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:11.220917 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:11.222166 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:11.222424 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:11.223730 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:11.223743 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:11.223769 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:50:11.224695 (+ 926us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:11.224849 (+ 154us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:50:11.224883 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:11.225048 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:11.225308 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 16:50:11.225739 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":409,"threads_started":1}
I20260928 16:50:11.231216 14861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12fa6a7b1a9c4a728ef024bc09a101eb: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:11.232385 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:50:11.232924 14861 log.cc:824] T 00000000000000000000000000000000 P 12fa6a7b1a9c4a728ef024bc09a101eb: Log is configured to *not* fsync() on all Append() calls
I20260928 16:50:11.237007 14861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12fa6a7b1a9c4a728ef024bc09a101eb: No bootstrap required, opened a new log
I20260928 16:50:11.246052 14861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12fa6a7b1a9c4a728ef024bc09a101eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fa6a7b1a9c4a728ef024bc09a101eb" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33507 } }
I20260928 16:50:11.246579 14861 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12fa6a7b1a9c4a728ef024bc09a101eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:11.246703 14861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12fa6a7b1a9c4a728ef024bc09a101eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12fa6a7b1a9c4a728ef024bc09a101eb, State: Initialized, Role: FOLLOWER
I20260928 16:50:11.247578 14861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12fa6a7b1a9c4a728ef024bc09a101eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fa6a7b1a9c4a728ef024bc09a101eb" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33507 } }
I20260928 16:50:11.247886 14861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12fa6a7b1a9c4a728ef024bc09a101eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:50:11.248054 14861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12fa6a7b1a9c4a728ef024bc09a101eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:50:11.248255 14861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12fa6a7b1a9c4a728ef024bc09a101eb [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:11.251093 14861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12fa6a7b1a9c4a728ef024bc09a101eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fa6a7b1a9c4a728ef024bc09a101eb" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33507 } }
I20260928 16:50:11.251828 14861 leader_election.cc:302] T 00000000000000000000000000000000 P 12fa6a7b1a9c4a728ef024bc09a101eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12fa6a7b1a9c4a728ef024bc09a101eb; no voters:
I20260928 16:50:11.252537 14861 leader_election.cc:288] T 00000000000000000000000000000000 P 12fa6a7b1a9c4a728ef024bc09a101eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:50:11.252719 14866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12fa6a7b1a9c4a728ef024bc09a101eb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:50:11.253441 14866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12fa6a7b1a9c4a728ef024bc09a101eb [term 1 LEADER]: Becoming Leader. State: Replica: 12fa6a7b1a9c4a728ef024bc09a101eb, State: Running, Role: LEADER
I20260928 16:50:11.254029 14866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12fa6a7b1a9c4a728ef024bc09a101eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fa6a7b1a9c4a728ef024bc09a101eb" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33507 } }
I20260928 16:50:11.254974 14861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12fa6a7b1a9c4a728ef024bc09a101eb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:50:11.257202 14868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12fa6a7b1a9c4a728ef024bc09a101eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12fa6a7b1a9c4a728ef024bc09a101eb. Latest consensus state: current_term: 1 leader_uuid: "12fa6a7b1a9c4a728ef024bc09a101eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fa6a7b1a9c4a728ef024bc09a101eb" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33507 } } }
I20260928 16:50:11.257602 14868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12fa6a7b1a9c4a728ef024bc09a101eb [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:11.258025 14867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12fa6a7b1a9c4a728ef024bc09a101eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12fa6a7b1a9c4a728ef024bc09a101eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fa6a7b1a9c4a728ef024bc09a101eb" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33507 } } }
I20260928 16:50:11.258313 14867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12fa6a7b1a9c4a728ef024bc09a101eb [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:11.259946 14872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:50:11.265846 14872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:50:11.276583 14872 catalog_manager.cc:1347] Generated new cluster ID: 034086fa71f4424a9b6874a68b02dbcd
I20260928 16:50:11.276790 14872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:50:11.294655 14872 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:50:11.296308 14872 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:50:11.307224 14872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12fa6a7b1a9c4a728ef024bc09a101eb: Generated new TSK 0
I20260928 16:50:11.308444 14872 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 16:50:11.359953 14891 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 16:50:11.360196 14891 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 16:50:11.362993 14864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:11.337436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50572 (local address 127.9.93.62:33507)
0928 16:50:11.337659 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:11.337670 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:11.338167 (+ 497us) server_negotiation.cc:407] Connection header received
0928 16:50:11.338589 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:11.338596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:11.338706 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:11.338885 (+ 179us) server_negotiation.cc:226] Negotiated authn=JWT
0928 16:50:11.339464 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:11.340633 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:11.342369 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:11.343101 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:11.343243 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:11.343948 (+ 705us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 16:50:11.362184 (+ 18236us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 16:50:11.362647 (+ 463us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50572: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 16:50:11.363147 14864 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50572: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 16:50:11.364770 14890 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:11.337257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33507 (local address 127.0.0.1:50572)
0928 16:50:11.338048 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:11.338068 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:11.338252 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:11.338920 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:11.338924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:11.338936 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0928 16:50:11.339263 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:11.339272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:11.340837 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:11.340841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:11.342213 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:11.342228 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:11.343238 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:11.343328 (+ 90us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:11.343536 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:11.343617 (+ 81us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 16:50:11.362555 (+ 18938us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 16:50:11.362874 (+ 319us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.93.62:33507: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":418,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.9.93.62:33507: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 16:50:11.366961 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 14795
I20260928 16:50:11.368361 14856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:11.465768 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 14795
2026-09-28T16:50:11Z chronyd exiting
I20260928 16:50:11.514680 9588 test_util.cc:175] -----------------------------------------------
I20260928 16:50:11.514834 9588 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790614166338986-9588-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (859 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 16:50:11.518833 9588 mini_oidc.cc:152] Starting JWKS server
I20260928 16:50:11.520871 9588 webserver.cc:460] Webserver started at https://127.0.0.1:34747/ using document root <none> and password file <none>
I20260928 16:50:11.521088 9588 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 16:50:11.521757 9588 webserver.cc:460] Webserver started at http://127.0.0.1:44761/ using document root <none> and password file <none>
2026-09-28T16:50:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:50:11Z Disabled control of system clock
I20260928 16:50:11.553169 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:33429
--webserver_interface=127.9.93.62
--webserver_port=0
--builtin_ntp_servers=127.9.93.20:36685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:33429
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34747/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790614166338986-9588-0/testchainca.cert with env {}
W20260928 16:50:11.915665 14898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:11.916023 14898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:11.916167 14898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:11.924022 14898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:50:11.924177 14898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:11.924252 14898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:11.924333 14898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:50:11.924396 14898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 16:50:11.924486 14898 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 16:50:11.924538 14898 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34747/jwks.json
I20260928 16:50:11.935377 14898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:36685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:33429
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:33429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34747/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790614166338986-9588-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:11.936954 14898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:11.938516 14898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:11.952273 14904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:11.952220 14906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:11.952246 14903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:11.953150 14898 server_base.cc:1034] running on GCE node
I20260928 16:50:11.954187 14898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:11.955962 14898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:11.957230 14898 hybrid_clock.cc:648] HybridClock initialized: now 1790614211957154 us; error 63 us; skew 500 ppm
I20260928 16:50:11.957736 14898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:50:11.960466 14898 webserver.cc:460] Webserver started at http://127.9.93.62:39951/ using document root <none> and password file <none>
I20260928 16:50:11.961300 14898 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:11.961421 14898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:11.961777 14898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:11.965126 14898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "65acff75bdb048cfabb3b4845ca94c81"
format_stamp: "Formatted at 2026-09-28 16:50:11 on dist-test-slave-wzcb"
server_key: "d2a15e37bcf892b5bc4abd0859f60ac1"
server_key_iv: "935a1cdb422b7fdb0072216fe85ec63a"
server_key_version: "encryptionkey@0"
I20260928 16:50:11.966003 14898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "65acff75bdb048cfabb3b4845ca94c81"
format_stamp: "Formatted at 2026-09-28 16:50:11 on dist-test-slave-wzcb"
server_key: "d2a15e37bcf892b5bc4abd0859f60ac1"
server_key_iv: "935a1cdb422b7fdb0072216fe85ec63a"
server_key_version: "encryptionkey@0"
I20260928 16:50:11.971941 14898 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 16:50:11.976022 14912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:11.977761 14898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:50:11.977973 14898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "65acff75bdb048cfabb3b4845ca94c81"
format_stamp: "Formatted at 2026-09-28 16:50:11 on dist-test-slave-wzcb"
server_key: "d2a15e37bcf892b5bc4abd0859f60ac1"
server_key_iv: "935a1cdb422b7fdb0072216fe85ec63a"
server_key_version: "encryptionkey@0"
I20260928 16:50:11.978217 14898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:11.999840 14898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:12.000869 14898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:12.001191 14898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:12.021134 14898 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:33429
I20260928 16:50:12.021142 14963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:33429 every 8 connection(s)
I20260928 16:50:12.022861 14898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:50:12.026881 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 14898
I20260928 16:50:12.027056 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:50:12.027776 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:12.027817 9588 external_mini_cluster.cc:1428] Setting key f88b741d96d2b89f9660972273dc20eb
I20260928 16:50:12.046442 14964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65acff75bdb048cfabb3b4845ca94c81: Bootstrap starting.
I20260928 16:50:12.046519 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:12.030153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51974 (local address 127.9.93.62:33429)
0928 16:50:12.031620 (+ 1467us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:12.031646 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:12.031712 (+ 66us) server_negotiation.cc:407] Connection header received
0928 16:50:12.033811 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:12.033857 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:12.034704 (+ 847us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:12.035359 (+ 655us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:12.036103 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:12.038291 (+ 2188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:12.039937 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:12.041140 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:12.041389 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:12.042744 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:12.042762 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:12.042793 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:50:12.043772 (+ 979us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:12.043924 (+ 152us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:50:12.043959 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:12.044102 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:12.044415 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 16:50:12.044880 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":477,"threads_started":1}
I20260928 16:50:12.050900 14964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65acff75bdb048cfabb3b4845ca94c81: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:12.051286 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:50:12.052363 14964 log.cc:824] T 00000000000000000000000000000000 P 65acff75bdb048cfabb3b4845ca94c81: Log is configured to *not* fsync() on all Append() calls
I20260928 16:50:12.056248 14964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65acff75bdb048cfabb3b4845ca94c81: No bootstrap required, opened a new log
I20260928 16:50:12.064569 14964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65acff75bdb048cfabb3b4845ca94c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65acff75bdb048cfabb3b4845ca94c81" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33429 } }
I20260928 16:50:12.065032 14964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65acff75bdb048cfabb3b4845ca94c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:12.065143 14964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65acff75bdb048cfabb3b4845ca94c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65acff75bdb048cfabb3b4845ca94c81, State: Initialized, Role: FOLLOWER
I20260928 16:50:12.065814 14964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65acff75bdb048cfabb3b4845ca94c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65acff75bdb048cfabb3b4845ca94c81" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33429 } }
I20260928 16:50:12.066058 14964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65acff75bdb048cfabb3b4845ca94c81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:50:12.066201 14964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65acff75bdb048cfabb3b4845ca94c81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:50:12.066387 14964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65acff75bdb048cfabb3b4845ca94c81 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:12.069176 14964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65acff75bdb048cfabb3b4845ca94c81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65acff75bdb048cfabb3b4845ca94c81" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33429 } }
I20260928 16:50:12.069783 14964 leader_election.cc:302] T 00000000000000000000000000000000 P 65acff75bdb048cfabb3b4845ca94c81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65acff75bdb048cfabb3b4845ca94c81; no voters:
I20260928 16:50:12.070428 14964 leader_election.cc:288] T 00000000000000000000000000000000 P 65acff75bdb048cfabb3b4845ca94c81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:50:12.070605 14969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65acff75bdb048cfabb3b4845ca94c81 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:50:12.071292 14969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65acff75bdb048cfabb3b4845ca94c81 [term 1 LEADER]: Becoming Leader. State: Replica: 65acff75bdb048cfabb3b4845ca94c81, State: Running, Role: LEADER
I20260928 16:50:12.071919 14969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65acff75bdb048cfabb3b4845ca94c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65acff75bdb048cfabb3b4845ca94c81" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33429 } }
I20260928 16:50:12.072718 14964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65acff75bdb048cfabb3b4845ca94c81 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:50:12.074833 14971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65acff75bdb048cfabb3b4845ca94c81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65acff75bdb048cfabb3b4845ca94c81. Latest consensus state: current_term: 1 leader_uuid: "65acff75bdb048cfabb3b4845ca94c81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65acff75bdb048cfabb3b4845ca94c81" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33429 } } }
I20260928 16:50:12.075347 14971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65acff75bdb048cfabb3b4845ca94c81 [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:12.075510 14970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65acff75bdb048cfabb3b4845ca94c81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65acff75bdb048cfabb3b4845ca94c81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65acff75bdb048cfabb3b4845ca94c81" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33429 } } }
I20260928 16:50:12.075822 14970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65acff75bdb048cfabb3b4845ca94c81 [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:12.079175 14975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:50:12.085155 14975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:50:12.096200 14975 catalog_manager.cc:1347] Generated new cluster ID: 06a5a74bb9174e56a7abee1dcb5baac6
I20260928 16:50:12.096356 14975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:50:12.119832 14975 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:50:12.121706 14975 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:50:12.137394 14975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65acff75bdb048cfabb3b4845ca94c81: Generated new TSK 0
I20260928 16:50:12.138422 14975 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 16:50:12.193849 14994 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 16:50:12.194025 14994 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 16:50:12.195142 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:12.166231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51980 (local address 127.9.93.62:33429)
0928 16:50:12.166414 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:12.166419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:12.166857 (+ 438us) server_negotiation.cc:407] Connection header received
0928 16:50:12.167167 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:12.167172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:12.167255 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:12.167435 (+ 180us) server_negotiation.cc:226] Negotiated authn=JWT
0928 16:50:12.168015 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:12.168838 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:12.170983 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:12.171578 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:12.171698 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:12.172415 (+ 717us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 16:50:12.194470 (+ 22055us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34747/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 16:50:12.194857 (+ 387us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51980: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34747/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 16:50:12.195290 14967 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51980: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34747/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 16:50:12.195631 14993 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:12.166025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33429 (local address 127.0.0.1:51980)
0928 16:50:12.166726 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:12.166754 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:12.166936 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:12.167475 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:12.167481 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:12.167495 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0928 16:50:12.167844 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:12.167857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:12.169001 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:12.169007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:12.170775 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:12.170798 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:12.171720 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:12.171802 (+ 82us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:12.172028 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:12.172082 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 16:50:12.194789 (+ 22707us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34747/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 16:50:12.195139 (+ 350us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.93.62:33429: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34747/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":373,"threads_started":1}
I20260928 16:50:12.196918 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 14898
I20260928 16:50:12.198429 14959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:12.297076 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 14898
2026-09-28T16:50:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (832 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 16:50:12.351778 9588 mini_oidc.cc:152] Starting JWKS server
I20260928 16:50:12.353945 9588 webserver.cc:460] Webserver started at https://127.0.0.1:37147/ using document root <none> and password file <none>
I20260928 16:50:12.354255 9588 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 16:50:12.354993 9588 webserver.cc:460] Webserver started at http://127.0.0.1:37529/ using document root <none> and password file <none>
2026-09-28T16:50:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:50:12Z Disabled control of system clock
I20260928 16:50:12.386886 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:44317
--webserver_interface=127.9.93.62
--webserver_port=0
--builtin_ntp_servers=127.9.93.20:42305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:44317
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37147/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 16:50:12.759168 15001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:12.759598 15001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:12.759706 15001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:12.767412 15001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:50:12.767552 15001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:12.767589 15001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:12.767664 15001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:50:12.767709 15001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 16:50:12.767772 15001 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 16:50:12.767818 15001 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37147/jwks.json
I20260928 16:50:12.777729 15001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:42305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:44317
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:44317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37147/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:12.779224 15001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:12.781446 15001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:12.794953 15009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:12.795035 15007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:12.795064 15006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:12.796227 15001 server_base.cc:1034] running on GCE node
I20260928 16:50:12.797215 15001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:12.798987 15001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:12.800211 15001 hybrid_clock.cc:648] HybridClock initialized: now 1790614212800132 us; error 73 us; skew 500 ppm
I20260928 16:50:12.800627 15001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:50:12.803467 15001 webserver.cc:460] Webserver started at http://127.9.93.62:44103/ using document root <none> and password file <none>
I20260928 16:50:12.804342 15001 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:12.804448 15001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:12.804791 15001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:12.808287 15001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "55b5dadea2a448b18ba837c52beb9cb0"
format_stamp: "Formatted at 2026-09-28 16:50:12 on dist-test-slave-wzcb"
server_key: "c89fb45aebab49aaa4aea85d3db6cc00"
server_key_iv: "b18b9eecff12169673dffd89e51061c2"
server_key_version: "encryptionkey@0"
I20260928 16:50:12.809238 15001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "55b5dadea2a448b18ba837c52beb9cb0"
format_stamp: "Formatted at 2026-09-28 16:50:12 on dist-test-slave-wzcb"
server_key: "c89fb45aebab49aaa4aea85d3db6cc00"
server_key_iv: "b18b9eecff12169673dffd89e51061c2"
server_key_version: "encryptionkey@0"
I20260928 16:50:12.815440 15001 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260928 16:50:12.819823 15015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:12.821681 15001 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:50:12.821902 15001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "55b5dadea2a448b18ba837c52beb9cb0"
format_stamp: "Formatted at 2026-09-28 16:50:12 on dist-test-slave-wzcb"
server_key: "c89fb45aebab49aaa4aea85d3db6cc00"
server_key_iv: "b18b9eecff12169673dffd89e51061c2"
server_key_version: "encryptionkey@0"
I20260928 16:50:12.822151 15001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:12.840988 15001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:12.842052 15001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:12.842391 15001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:12.861630 15001 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:44317
I20260928 16:50:12.861649 15066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:44317 every 8 connection(s)
I20260928 16:50:12.863193 15001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:50:12.867738 15067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:12.871661 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 15001
I20260928 16:50:12.871881 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:50:12.872557 9588 external_mini_cluster.cc:1428] Setting key e2b59e70c18163808e848277179ce62a
I20260928 16:50:12.882373 15067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55b5dadea2a448b18ba837c52beb9cb0: Bootstrap starting.
I20260928 16:50:12.886571 15067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55b5dadea2a448b18ba837c52beb9cb0: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:12.888017 15067 log.cc:824] T 00000000000000000000000000000000 P 55b5dadea2a448b18ba837c52beb9cb0: Log is configured to *not* fsync() on all Append() calls
I20260928 16:50:12.888664 15070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:12.874801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51018 (local address 127.9.93.62:44317)
0928 16:50:12.875977 (+ 1176us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:12.875990 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:12.876040 (+ 50us) server_negotiation.cc:407] Connection header received
0928 16:50:12.878096 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:12.878130 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:12.878804 (+ 674us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:12.879332 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:12.880271 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:12.881811 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:12.883464 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:12.884273 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:12.884514 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:12.885764 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:12.885773 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:12.885792 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:50:12.886405 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:12.886527 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:50:12.886552 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:12.886685 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:12.886933 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 16:50:12.887363 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":409,"threads_started":1}
I20260928 16:50:12.891758 15067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55b5dadea2a448b18ba837c52beb9cb0: No bootstrap required, opened a new log
I20260928 16:50:12.893424 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:50:12.900738 15067 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55b5dadea2a448b18ba837c52beb9cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b5dadea2a448b18ba837c52beb9cb0" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 44317 } }
I20260928 16:50:12.901244 15067 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55b5dadea2a448b18ba837c52beb9cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:12.901363 15067 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55b5dadea2a448b18ba837c52beb9cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55b5dadea2a448b18ba837c52beb9cb0, State: Initialized, Role: FOLLOWER
I20260928 16:50:12.902082 15067 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55b5dadea2a448b18ba837c52beb9cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b5dadea2a448b18ba837c52beb9cb0" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 44317 } }
I20260928 16:50:12.902346 15067 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55b5dadea2a448b18ba837c52beb9cb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:50:12.902511 15067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55b5dadea2a448b18ba837c52beb9cb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:50:12.902711 15067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55b5dadea2a448b18ba837c52beb9cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:12.905282 15067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55b5dadea2a448b18ba837c52beb9cb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b5dadea2a448b18ba837c52beb9cb0" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 44317 } }
I20260928 16:50:12.905901 15067 leader_election.cc:302] T 00000000000000000000000000000000 P 55b5dadea2a448b18ba837c52beb9cb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55b5dadea2a448b18ba837c52beb9cb0; no voters:
I20260928 16:50:12.906597 15067 leader_election.cc:288] T 00000000000000000000000000000000 P 55b5dadea2a448b18ba837c52beb9cb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:50:12.906783 15072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55b5dadea2a448b18ba837c52beb9cb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:50:12.908192 15072 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55b5dadea2a448b18ba837c52beb9cb0 [term 1 LEADER]: Becoming Leader. State: Replica: 55b5dadea2a448b18ba837c52beb9cb0, State: Running, Role: LEADER
I20260928 16:50:12.908928 15072 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55b5dadea2a448b18ba837c52beb9cb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b5dadea2a448b18ba837c52beb9cb0" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 44317 } }
I20260928 16:50:12.909300 15067 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55b5dadea2a448b18ba837c52beb9cb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:50:12.912120 15074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b5dadea2a448b18ba837c52beb9cb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55b5dadea2a448b18ba837c52beb9cb0. Latest consensus state: current_term: 1 leader_uuid: "55b5dadea2a448b18ba837c52beb9cb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b5dadea2a448b18ba837c52beb9cb0" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 44317 } } }
I20260928 16:50:12.912576 15074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b5dadea2a448b18ba837c52beb9cb0 [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:12.912993 15073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b5dadea2a448b18ba837c52beb9cb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55b5dadea2a448b18ba837c52beb9cb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b5dadea2a448b18ba837c52beb9cb0" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 44317 } } }
I20260928 16:50:12.913326 15073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b5dadea2a448b18ba837c52beb9cb0 [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:12.916194 15082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:50:12.921442 15082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:50:12.932574 15082 catalog_manager.cc:1347] Generated new cluster ID: 779cf5886089423a8a6eb77d51a42fcc
I20260928 16:50:12.932723 15082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:50:12.954363 15082 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:50:12.956791 15082 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:50:12.971936 15082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55b5dadea2a448b18ba837c52beb9cb0: Generated new TSK 0
I20260928 16:50:12.972997 15082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:50:12.998807 15070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:12.989006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51022 (local address 127.9.93.62:44317)
0928 16:50:12.989217 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:12.989224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:12.989660 (+ 436us) server_negotiation.cc:407] Connection header received
0928 16:50:12.989909 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:12.989916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:12.990037 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:12.990259 (+ 222us) server_negotiation.cc:226] Negotiated authn=JWT
0928 16:50:12.990799 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:12.992038 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:12.993711 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:12.994439 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:12.994587 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:12.995222 (+ 635us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 16:50:12.998025 (+ 2803us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37147/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 16:50:12.998453 (+ 428us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51022: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37147/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":76}
W20260928 16:50:12.998979 15070 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51022: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37147/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 16:50:13.000393 15096 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:12.988808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:44317 (local address 127.0.0.1:51022)
0928 16:50:12.989541 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:12.989562 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:12.989718 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:12.990300 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:12.990305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:12.990317 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0928 16:50:12.990620 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:12.990630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:12.992256 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:12.992261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:12.993527 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:12.993545 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:12.994613 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:12.994713 (+ 100us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:12.994872 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:12.994917 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 16:50:12.998343 (+ 3426us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37147/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 16:50:12.999892 (+ 1549us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.93.62:44317: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37147/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":1145,"thread_start_us":400,"threads_started":1}
I20260928 16:50:13.002683 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 15001
I20260928 16:50:13.004192 15062 generic_service.cc:196] Checking for leaks (request via RPC)
E20260928 16:50:13.013289 15097 webserver.cc:533] Webserver: sslize failed:
E20260928 16:50:13.013460 15097 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 16:50:13.101958 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 15001
2026-09-28T16:50:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (804 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T16:50:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:50:13Z Disabled control of system clock
I20260928 16:50:13.187196 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:39081
--webserver_interface=127.9.93.62
--webserver_port=0
--builtin_ntp_servers=127.9.93.20:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:39081
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790614166338986-9588-0/testchainca.cert with env {}
W20260928 16:50:13.551555 15102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:13.551913 15102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:13.552037 15102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:13.560055 15102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:50:13.560194 15102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:13.560236 15102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:13.560322 15102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:50:13.560391 15102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 16:50:13.560490 15102 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 16:50:13.560550 15102 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 16:50:13.571774 15102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:39081
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:39081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790614166338986-9588-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:13.573429 15102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:13.574996 15102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:13.588371 15110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:13.588512 15107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:13.588548 15108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:13.589565 15102 server_base.cc:1034] running on GCE node
I20260928 16:50:13.590566 15102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:13.592468 15102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:13.593751 15102 hybrid_clock.cc:648] HybridClock initialized: now 1790614213593688 us; error 53 us; skew 500 ppm
I20260928 16:50:13.594218 15102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:50:13.607573 15102 webserver.cc:460] Webserver started at http://127.9.93.62:34477/ using document root <none> and password file <none>
I20260928 16:50:13.608497 15102 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:13.608619 15102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:13.608979 15102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:13.612500 15102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "5b107d1eb8ad470a8b5e61054ed5cfe1"
format_stamp: "Formatted at 2026-09-28 16:50:13 on dist-test-slave-wzcb"
server_key: "bc537ff2e1fa885400ee5785b962aa72"
server_key_iv: "bc75fe4498ec21b0c26bbde3cf35d431"
server_key_version: "encryptionkey@0"
I20260928 16:50:13.613487 15102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "5b107d1eb8ad470a8b5e61054ed5cfe1"
format_stamp: "Formatted at 2026-09-28 16:50:13 on dist-test-slave-wzcb"
server_key: "bc537ff2e1fa885400ee5785b962aa72"
server_key_iv: "bc75fe4498ec21b0c26bbde3cf35d431"
server_key_version: "encryptionkey@0"
I20260928 16:50:13.619859 15102 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 16:50:13.624271 15116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:13.625985 15102 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:50:13.626209 15102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "5b107d1eb8ad470a8b5e61054ed5cfe1"
format_stamp: "Formatted at 2026-09-28 16:50:13 on dist-test-slave-wzcb"
server_key: "bc537ff2e1fa885400ee5785b962aa72"
server_key_iv: "bc75fe4498ec21b0c26bbde3cf35d431"
server_key_version: "encryptionkey@0"
I20260928 16:50:13.626451 15102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:13.653975 15102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:13.655026 15102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:13.655392 15102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:13.675127 15102 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:39081
I20260928 16:50:13.675145 15167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:39081 every 8 connection(s)
I20260928 16:50:13.677188 15102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:50:13.682359 15168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:13.683007 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 15102
I20260928 16:50:13.683230 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:50:13.684165 9588 external_mini_cluster.cc:1428] Setting key 967955d8cbd0a27e2ac47daf93488058
I20260928 16:50:13.696683 15168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b107d1eb8ad470a8b5e61054ed5cfe1: Bootstrap starting.
I20260928 16:50:13.701467 15168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b107d1eb8ad470a8b5e61054ed5cfe1: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:13.703222 15168 log.cc:824] T 00000000000000000000000000000000 P 5b107d1eb8ad470a8b5e61054ed5cfe1: Log is configured to *not* fsync() on all Append() calls
I20260928 16:50:13.706943 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:13.687057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48766 (local address 127.9.93.62:39081)
0928 16:50:13.692278 (+ 5221us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:13.692295 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:13.692356 (+ 61us) server_negotiation.cc:407] Connection header received
0928 16:50:13.694424 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:13.694461 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:13.695132 (+ 671us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:13.695640 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:13.696472 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:13.698556 (+ 2084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:13.700900 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:13.701890 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:13.702113 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:13.703714 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:13.703724 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:13.703745 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:50:13.704687 (+ 942us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:13.704824 (+ 137us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:50:13.704867 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:13.705025 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:13.705335 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 16:50:13.705749 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4632,"thread_start_us":4238,"threads_started":1}
I20260928 16:50:13.707764 15168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b107d1eb8ad470a8b5e61054ed5cfe1: No bootstrap required, opened a new log
I20260928 16:50:13.711921 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:50:13.717458 15168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b107d1eb8ad470a8b5e61054ed5cfe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b107d1eb8ad470a8b5e61054ed5cfe1" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39081 } }
I20260928 16:50:13.717943 15168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b107d1eb8ad470a8b5e61054ed5cfe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:13.718035 15168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b107d1eb8ad470a8b5e61054ed5cfe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b107d1eb8ad470a8b5e61054ed5cfe1, State: Initialized, Role: FOLLOWER
I20260928 16:50:13.718703 15168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b107d1eb8ad470a8b5e61054ed5cfe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b107d1eb8ad470a8b5e61054ed5cfe1" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39081 } }
I20260928 16:50:13.719005 15168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b107d1eb8ad470a8b5e61054ed5cfe1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:50:13.719169 15168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b107d1eb8ad470a8b5e61054ed5cfe1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:50:13.719424 15168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b107d1eb8ad470a8b5e61054ed5cfe1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:13.722072 15168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b107d1eb8ad470a8b5e61054ed5cfe1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b107d1eb8ad470a8b5e61054ed5cfe1" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39081 } }
I20260928 16:50:13.722730 15168 leader_election.cc:302] T 00000000000000000000000000000000 P 5b107d1eb8ad470a8b5e61054ed5cfe1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b107d1eb8ad470a8b5e61054ed5cfe1; no voters:
I20260928 16:50:13.723477 15168 leader_election.cc:288] T 00000000000000000000000000000000 P 5b107d1eb8ad470a8b5e61054ed5cfe1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:50:13.723693 15173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b107d1eb8ad470a8b5e61054ed5cfe1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:50:13.724787 15173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b107d1eb8ad470a8b5e61054ed5cfe1 [term 1 LEADER]: Becoming Leader. State: Replica: 5b107d1eb8ad470a8b5e61054ed5cfe1, State: Running, Role: LEADER
I20260928 16:50:13.725364 15173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b107d1eb8ad470a8b5e61054ed5cfe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b107d1eb8ad470a8b5e61054ed5cfe1" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39081 } }
I20260928 16:50:13.726013 15168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b107d1eb8ad470a8b5e61054ed5cfe1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:50:13.728159 15174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b107d1eb8ad470a8b5e61054ed5cfe1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b107d1eb8ad470a8b5e61054ed5cfe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b107d1eb8ad470a8b5e61054ed5cfe1" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39081 } } }
I20260928 16:50:13.728488 15174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b107d1eb8ad470a8b5e61054ed5cfe1 [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:13.729269 15175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b107d1eb8ad470a8b5e61054ed5cfe1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b107d1eb8ad470a8b5e61054ed5cfe1. Latest consensus state: current_term: 1 leader_uuid: "5b107d1eb8ad470a8b5e61054ed5cfe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b107d1eb8ad470a8b5e61054ed5cfe1" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39081 } } }
I20260928 16:50:13.729696 15175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b107d1eb8ad470a8b5e61054ed5cfe1 [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:13.734846 15179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:50:13.740224 15179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:50:13.749675 15179 catalog_manager.cc:1347] Generated new cluster ID: fdb64f60b5634ab783dc9add1b5572fa
I20260928 16:50:13.749817 15179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:50:13.772241 15179 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:50:13.774178 15179 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:50:13.783048 15179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b107d1eb8ad470a8b5e61054ed5cfe1: Generated new TSK 0
I20260928 16:50:13.784339 15179 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:50:13.844254 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:13.800047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48770 (local address 127.9.93.62:39081)
0928 16:50:13.800280 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:13.800287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:13.800700 (+ 413us) server_negotiation.cc:407] Connection header received
0928 16:50:13.800963 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:13.800970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:13.801080 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:13.801278 (+ 198us) server_negotiation.cc:226] Negotiated authn=JWT
0928 16:50:13.801724 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:13.802890 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:13.804449 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:13.805177 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:13.805325 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:13.805887 (+ 562us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 16:50:13.843359 (+ 37472us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 16:50:13.843907 (+ 548us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48770: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":78}
W20260928 16:50:13.844564 15197 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:13.799924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:39081 (local address 127.0.0.1:48770)
0928 16:50:13.800584 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:13.800605 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:13.800751 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:13.801271 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:13.801275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:13.801285 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0928 16:50:13.801574 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:13.801582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:13.803122 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:13.803127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:13.804280 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:13.804295 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:13.805307 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:13.805365 (+ 58us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:13.805520 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:13.805561 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 16:50:13.843771 (+ 38210us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 16:50:13.844108 (+ 337us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.93.62:39081: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":357,"threads_started":1}
W20260928 16:50:13.844548 15171 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48770: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260928 16:50:13.846503 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 15102
I20260928 16:50:13.847893 15163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:13.947259 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 15102
2026-09-28T16:50:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (825 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (445 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (451 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:50:14 dist-test-slave-wzcb krb5kdc[15212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:50:14 dist-test-slave-wzcb krb5kdc[15212](info): set up 2 sockets
Sep 28 16:50:14 dist-test-slave-wzcb krb5kdc[15212](info): commencing operation
krb5kdc: starting...
W20260928 16:50:15.870378 9588 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:50:16 dist-test-slave-wzcb krb5kdc[15212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:50:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:50:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:50:16.104250 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:33441
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:39773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:33441
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:50:16.446264 15225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:16.446616 15225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:16.446744 15225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:16.454275 15225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:50:16.454409 15225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:16.454478 15225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:16.454562 15225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:50:16.454622 15225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:50:16.465292 15225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:33441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:33441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:16.466845 15225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:16.468364 15225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:16.480706 15230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:16.480746 15231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:16.480674 15233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:16.481592 15225 server_base.cc:1034] running on GCE node
I20260928 16:50:16.482501 15225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:16.484181 15225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:16.485404 15225 hybrid_clock.cc:648] HybridClock initialized: now 1790614216485340 us; error 52 us; skew 500 ppm
I20260928 16:50:16.485793 15225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:16 dist-test-slave-wzcb krb5kdc[15212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614216, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:16.488957 15225 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:50:16.490454 15225 webserver.cc:460] Webserver started at http://127.9.93.62:33543/ using document root <none> and password file <none>
I20260928 16:50:16.491185 15225 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:16.491333 15225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:16.491662 15225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:16.494881 15225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "d67f917b8f944993b9378df12f9c0ab0"
format_stamp: "Formatted at 2026-09-28 16:50:16 on dist-test-slave-wzcb"
server_key: "8b3ce22c623983da8c7b15bf39c1085f"
server_key_iv: "f7c2ef4768ac4db3edf233bc570a4039"
server_key_version: "encryptionkey@0"
I20260928 16:50:16.495822 15225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "d67f917b8f944993b9378df12f9c0ab0"
format_stamp: "Formatted at 2026-09-28 16:50:16 on dist-test-slave-wzcb"
server_key: "8b3ce22c623983da8c7b15bf39c1085f"
server_key_iv: "f7c2ef4768ac4db3edf233bc570a4039"
server_key_version: "encryptionkey@0"
I20260928 16:50:16.501562 15225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:50:16.505647 15240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:16.507274 15225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:50:16.507529 15225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "d67f917b8f944993b9378df12f9c0ab0"
format_stamp: "Formatted at 2026-09-28 16:50:16 on dist-test-slave-wzcb"
server_key: "8b3ce22c623983da8c7b15bf39c1085f"
server_key_iv: "f7c2ef4768ac4db3edf233bc570a4039"
server_key_version: "encryptionkey@0"
I20260928 16:50:16.507748 15225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:16.521286 15225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:16.525924 15225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:16.526252 15225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:16.544145 15225 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:33441
I20260928 16:50:16.544173 15291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:33441 every 8 connection(s)
I20260928 16:50:16.545648 15225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:50:16.547053 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 15225
I20260928 16:50:16.547287 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:50:16.548159 9588 external_mini_cluster.cc:1428] Setting key a116c8064813a9f0a6513f9513eb2275
I20260928 16:50:16.550875 15292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 16:50:16 dist-test-slave-wzcb krb5kdc[15212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:50:16.562410 15292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d67f917b8f944993b9378df12f9c0ab0: Bootstrap starting.
I20260928 16:50:16.566066 15292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d67f917b8f944993b9378df12f9c0ab0: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:16.567210 15292 log.cc:824] T 00000000000000000000000000000000 P d67f917b8f944993b9378df12f9c0ab0: Log is configured to *not* fsync() on all Append() calls
I20260928 16:50:16.570612 15292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d67f917b8f944993b9378df12f9c0ab0: No bootstrap required, opened a new log
I20260928 16:50:16.570675 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:16.550411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34378 (local address 127.9.93.62:33441)
0928 16:50:16.551226 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:16.551239 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:16.551283 (+ 44us) server_negotiation.cc:407] Connection header received
0928 16:50:16.553021 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:16.553046 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:16.553473 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:16.553886 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:16.555367 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:16.556764 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:16.558062 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:16.558771 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:16.558952 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:16.563202 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:16.563211 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:16.563228 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:16.563274 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:16.566291 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:16.566965 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:16.566972 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:16.566979 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:16.567183 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:16.567632 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:16.567636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:16.567638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:16.568432 (+ 794us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:16.568718 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:16.569205 (+ 487us) server_negotiation.cc:299] Negotiation successful
0928 16:50:16.569576 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":337,"threads_started":1}
I20260928 16:50:16.578819 15292 raft_consensus.cc:348] T 00000000000000000000000000000000 P d67f917b8f944993b9378df12f9c0ab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d67f917b8f944993b9378df12f9c0ab0" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33441 } }
I20260928 16:50:16.579226 15292 raft_consensus.cc:374] T 00000000000000000000000000000000 P d67f917b8f944993b9378df12f9c0ab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:16.579348 15292 raft_consensus.cc:729] T 00000000000000000000000000000000 P d67f917b8f944993b9378df12f9c0ab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d67f917b8f944993b9378df12f9c0ab0, State: Initialized, Role: FOLLOWER
I20260928 16:50:16.579988 15292 consensus_queue.cc:260] T 00000000000000000000000000000000 P d67f917b8f944993b9378df12f9c0ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d67f917b8f944993b9378df12f9c0ab0" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33441 } }
I20260928 16:50:16.580243 15292 raft_consensus.cc:388] T 00000000000000000000000000000000 P d67f917b8f944993b9378df12f9c0ab0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:50:16.580370 15292 raft_consensus.cc:482] T 00000000000000000000000000000000 P d67f917b8f944993b9378df12f9c0ab0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:50:16.580536 15292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d67f917b8f944993b9378df12f9c0ab0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:16.582721 15292 raft_consensus.cc:504] T 00000000000000000000000000000000 P d67f917b8f944993b9378df12f9c0ab0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d67f917b8f944993b9378df12f9c0ab0" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33441 } }
I20260928 16:50:16.583350 15292 leader_election.cc:302] T 00000000000000000000000000000000 P d67f917b8f944993b9378df12f9c0ab0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d67f917b8f944993b9378df12f9c0ab0; no voters:
I20260928 16:50:16.583969 15292 leader_election.cc:288] T 00000000000000000000000000000000 P d67f917b8f944993b9378df12f9c0ab0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:50:16.584115 15297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d67f917b8f944993b9378df12f9c0ab0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:50:16.584748 15297 raft_consensus.cc:686] T 00000000000000000000000000000000 P d67f917b8f944993b9378df12f9c0ab0 [term 1 LEADER]: Becoming Leader. State: Replica: d67f917b8f944993b9378df12f9c0ab0, State: Running, Role: LEADER
I20260928 16:50:16.585307 15297 consensus_queue.cc:237] T 00000000000000000000000000000000 P d67f917b8f944993b9378df12f9c0ab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d67f917b8f944993b9378df12f9c0ab0" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33441 } }
I20260928 16:50:16.586072 15292 sys_catalog.cc:564] T 00000000000000000000000000000000 P d67f917b8f944993b9378df12f9c0ab0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:50:16.588348 15298 sys_catalog.cc:455] T 00000000000000000000000000000000 P d67f917b8f944993b9378df12f9c0ab0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d67f917b8f944993b9378df12f9c0ab0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d67f917b8f944993b9378df12f9c0ab0" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33441 } } }
I20260928 16:50:16.588732 15298 sys_catalog.cc:458] T 00000000000000000000000000000000 P d67f917b8f944993b9378df12f9c0ab0 [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:16.588865 15299 sys_catalog.cc:455] T 00000000000000000000000000000000 P d67f917b8f944993b9378df12f9c0ab0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d67f917b8f944993b9378df12f9c0ab0. Latest consensus state: current_term: 1 leader_uuid: "d67f917b8f944993b9378df12f9c0ab0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d67f917b8f944993b9378df12f9c0ab0" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33441 } } }
I20260928 16:50:16.589197 15299 sys_catalog.cc:458] T 00000000000000000000000000000000 P d67f917b8f944993b9378df12f9c0ab0 [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:16.591383 15305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:50:16.595988 15305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:50:16.604784 15305 catalog_manager.cc:1347] Generated new cluster ID: f088f291ff2e45ec950a6a01630de67f
I20260928 16:50:16.604930 15305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:50:16.633606 15305 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:50:16.635105 15305 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:50:16.641072 15305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d67f917b8f944993b9378df12f9c0ab0: Generated new TSK 0
I20260928 16:50:16.642020 15305 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
I20260928 16:50:16.704218 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33441
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:39773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:50:17.061690 15318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:17.062042 15318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:17.062191 15318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:17.070215 15318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:17.070344 15318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:17.070518 15318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:50:17.081817 15318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:17.083349 15318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:17.084821 15318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:17.099793 15324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:17.099826 15326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:17.099751 15323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:17.100801 15318 server_base.cc:1034] running on GCE node
I20260928 16:50:17.101550 15318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:17.102690 15318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:17.103945 15318 hybrid_clock.cc:648] HybridClock initialized: now 1790614217103870 us; error 54 us; skew 500 ppm
I20260928 16:50:17.104357 15318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:17 dist-test-slave-wzcb krb5kdc[15212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614217, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:17.108033 15318 init.cc:377] Logged in from keytab as kudu/127.9.93.1@KRBTEST.COM (short username kudu)
I20260928 16:50:17.109829 15318 webserver.cc:460] Webserver started at http://127.9.93.1:45641/ using document root <none> and password file <none>
I20260928 16:50:17.110666 15318 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:17.110803 15318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:17.111155 15318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:17.114902 15318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "36862061a7eb4326902cad292b76135e"
format_stamp: "Formatted at 2026-09-28 16:50:17 on dist-test-slave-wzcb"
server_key: "eb003184a28f204da2f5be4560f70db8"
server_key_iv: "82a2a7090f9c1f013960eca98b13206c"
server_key_version: "encryptionkey@0"
I20260928 16:50:17.115842 15318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "36862061a7eb4326902cad292b76135e"
format_stamp: "Formatted at 2026-09-28 16:50:17 on dist-test-slave-wzcb"
server_key: "eb003184a28f204da2f5be4560f70db8"
server_key_iv: "82a2a7090f9c1f013960eca98b13206c"
server_key_version: "encryptionkey@0"
I20260928 16:50:17.122035 15318 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260928 16:50:17.126333 15333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:17.128080 15318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:50:17.128283 15318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "36862061a7eb4326902cad292b76135e"
format_stamp: "Formatted at 2026-09-28 16:50:17 on dist-test-slave-wzcb"
server_key: "eb003184a28f204da2f5be4560f70db8"
server_key_iv: "82a2a7090f9c1f013960eca98b13206c"
server_key_version: "encryptionkey@0"
I20260928 16:50:17.128515 15318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:17.149181 15318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:17.154098 15318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:17.154408 15318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:17.155633 15318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:50:17.157301 15318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:17.157392 15318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:17.157526 15318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:17.157581 15318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:17.191350 15318 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:37241
I20260928 16:50:17.191385 15445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:37241 every 8 connection(s)
I20260928 16:50:17.192778 15318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:50:17.198127 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 15318
I20260928 16:50:17.198364 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:50:17.199216 9588 external_mini_cluster.cc:1428] Setting key c12a1bae88a50a6788df946f4add2792
Sep 28 16:50:17 dist-test-slave-wzcb krb5kdc[15212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614217, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:50:17.217716 15448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:17.195468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:36495 (local address 127.9.93.62:33441)
0928 16:50:17.195934 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:17.195939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:17.197191 (+ 1252us) server_negotiation.cc:407] Connection header received
0928 16:50:17.200281 (+ 3090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:17.200288 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:17.200431 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:17.200555 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:17.203181 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:17.204281 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:17.206667 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:17.207114 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:17.207276 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:17.211299 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:17.211325 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:17.211331 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:17.211380 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:17.214138 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:17.215126 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:17.215130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:17.215132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:17.215257 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:17.215781 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:17.215786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:17.215788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:17.216103 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:17.216218 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:17.217242 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0928 16:50:17.217440 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":247,"threads_started":1}
I20260928 16:50:17.219434 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:17.196016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33441 (local address 127.9.93.1:36495)
0928 16:50:17.196944 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:17.197011 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:17.199784 (+ 2773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:17.200940 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:17.200955 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:17.201895 (+ 940us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:17.202942 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:17.202977 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:17.204467 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:17.204473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:17.206493 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:17.206509 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:17.207430 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:17.207439 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:17.207736 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:17.208429 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:17.208461 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:17.211017 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:17.214308 (+ 3291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:17.214319 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:17.214341 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:17.214991 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:17.215383 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:17.215388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:17.215391 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:17.215666 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:17.216247 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:17.216255 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:17.216816 (+ 561us) client_negotiation.cc:769] Sending connection context
0928 16:50:17.217160 (+ 344us) client_negotiation.cc:240] Negotiation successful
0928 16:50:17.217804 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":339,"threads_started":1}
I20260928 16:50:17.221205 15446 heartbeater.cc:347] Connected to a master server at 127.9.93.62:33441
I20260928 16:50:17.221581 15446 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:17.222672 15446 heartbeater.cc:511] Master 127.9.93.62:33441 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.93.2@KRBTEST.COM; defaulting to no policy
I20260928 16:50:17.225231 15257 ts_manager.cc:194] Registered new tserver with Master: 36862061a7eb4326902cad292b76135e (127.9.93.1:37241)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.2@KRBTEST.COM" created.
I20260928 16:50:17.226979 15257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.1@KRBTEST.COM'} at 127.9.93.1:36495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
I20260928 16:50:17.255419 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33441
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:39773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:50:17.613427 15452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:17.613869 15452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:17.614051 15452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:17.622999 15452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:17.623139 15452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:17.623358 15452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:50:17.634510 15452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:17.636086 15452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:17.637593 15452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:17.652444 15457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:17.652540 15460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:17.652679 15458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:17.653967 15452 server_base.cc:1034] running on GCE node
I20260928 16:50:17.654738 15452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:17.656095 15452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:17.657332 15452 hybrid_clock.cc:648] HybridClock initialized: now 1790614217657253 us; error 59 us; skew 500 ppm
I20260928 16:50:17.657753 15452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:17 dist-test-slave-wzcb krb5kdc[15212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614217, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:17.661264 15452 init.cc:377] Logged in from keytab as kudu/127.9.93.2@KRBTEST.COM (short username kudu)
I20260928 16:50:17.662925 15452 webserver.cc:460] Webserver started at http://127.9.93.2:42175/ using document root <none> and password file <none>
I20260928 16:50:17.663753 15452 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:17.663865 15452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:17.664160 15452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:17.667567 15452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "34049ba6da854d81871e4bfa81a768de"
format_stamp: "Formatted at 2026-09-28 16:50:17 on dist-test-slave-wzcb"
server_key: "824878b6542c2c5eaf65071d49a5eb79"
server_key_iv: "e2f68ae24944805bcc1fae819dd0a69d"
server_key_version: "encryptionkey@0"
I20260928 16:50:17.668416 15452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "34049ba6da854d81871e4bfa81a768de"
format_stamp: "Formatted at 2026-09-28 16:50:17 on dist-test-slave-wzcb"
server_key: "824878b6542c2c5eaf65071d49a5eb79"
server_key_iv: "e2f68ae24944805bcc1fae819dd0a69d"
server_key_version: "encryptionkey@0"
I20260928 16:50:17.674471 15452 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260928 16:50:17.678602 15467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:17.680245 15452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:50:17.680428 15452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "34049ba6da854d81871e4bfa81a768de"
format_stamp: "Formatted at 2026-09-28 16:50:17 on dist-test-slave-wzcb"
server_key: "824878b6542c2c5eaf65071d49a5eb79"
server_key_iv: "e2f68ae24944805bcc1fae819dd0a69d"
server_key_version: "encryptionkey@0"
I20260928 16:50:17.680649 15452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:17.714826 15452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:17.719852 15452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:17.720203 15452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:17.721335 15452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:50:17.722915 15452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:17.722999 15452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:17.723140 15452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:17.723198 15452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:17.756356 15452 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:45373
I20260928 16:50:17.756407 15579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:45373 every 8 connection(s)
I20260928 16:50:17.757845 15452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:50:17.758973 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 15452
I20260928 16:50:17.759177 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:50:17.759810 9588 external_mini_cluster.cc:1428] Setting key a862529c7e060674854f2d37638fc153
Sep 28 16:50:17 dist-test-slave-wzcb krb5kdc[15212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614217, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:50:17.782303 15582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:17.761039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:44225 (local address 127.9.93.62:33441)
0928 16:50:17.761488 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:17.761493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:17.762964 (+ 1471us) server_negotiation.cc:407] Connection header received
0928 16:50:17.765876 (+ 2912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:17.765881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:17.765959 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:17.766063 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:17.768426 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:17.769240 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:17.771487 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:17.771955 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:17.772075 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:17.776029 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:17.776034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:17.776037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:17.776077 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:17.778660 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:17.779622 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:17.779626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:17.779628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:17.779769 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:17.780233 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:17.780237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:17.780239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:17.780534 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:17.780700 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:17.781639 (+ 939us) server_negotiation.cc:299] Negotiation successful
0928 16:50:17.781895 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":238,"threads_started":1}
WARNING: no policy specified for kudu/127.9.93.3@KRBTEST.COM; defaulting to no policy
I20260928 16:50:17.783854 15584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:17.761809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33441 (local address 127.9.93.2:44225)
0928 16:50:17.762767 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:17.762827 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:17.765521 (+ 2694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:17.766423 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:17.766436 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:17.767262 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:17.768205 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:17.768233 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:17.769419 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:17.769424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:17.771298 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:17.771342 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:17.772218 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:17.772226 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:17.772470 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:17.773192 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:17.773216 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:17.775742 (+ 2526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:17.778831 (+ 3089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:17.778842 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:17.778861 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:17.779479 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:17.779883 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:17.779888 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:17.779892 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:17.780115 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:17.780694 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:17.780703 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:17.781226 (+ 523us) client_negotiation.cc:769] Sending connection context
0928 16:50:17.781536 (+ 310us) client_negotiation.cc:240] Negotiation successful
0928 16:50:17.782199 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":387,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.3@KRBTEST.COM" created.
I20260928 16:50:17.785534 15580 heartbeater.cc:347] Connected to a master server at 127.9.93.62:33441
I20260928 16:50:17.785929 15580 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:17.786996 15580 heartbeater.cc:511] Master 127.9.93.62:33441 requested a full tablet report, sending...
I20260928 16:50:17.788923 15257 ts_manager.cc:194] Registered new tserver with Master: 34049ba6da854d81871e4bfa81a768de (127.9.93.2:45373)
I20260928 16:50:17.789841 15257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.2@KRBTEST.COM'} at 127.9.93.2:44225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
I20260928 16:50:17.813052 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33441
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:39773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:50:18.168033 15587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:18.168354 15587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:18.168501 15587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:18.176447 15587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:18.176587 15587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:18.176745 15587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:50:18.187745 15587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:18.189260 15587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:18.191031 15587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:18.206580 15592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:18.206655 15595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:18.206614 15593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:18.207629 15587 server_base.cc:1034] running on GCE node
I20260928 16:50:18.208302 15587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:18.209347 15587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:18.210553 15587 hybrid_clock.cc:648] HybridClock initialized: now 1790614218210497 us; error 40 us; skew 500 ppm
I20260928 16:50:18.210894 15587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:18 dist-test-slave-wzcb krb5kdc[15212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614218, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:18.214323 15587 init.cc:377] Logged in from keytab as kudu/127.9.93.3@KRBTEST.COM (short username kudu)
I20260928 16:50:18.215963 15587 webserver.cc:460] Webserver started at http://127.9.93.3:40535/ using document root <none> and password file <none>
I20260928 16:50:18.216724 15587 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:18.216840 15587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:18.217178 15587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:18.220458 15587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "fdad8e24cf234786b2edce754c78d9ef"
format_stamp: "Formatted at 2026-09-28 16:50:18 on dist-test-slave-wzcb"
server_key: "ee580012edb813ae8a929f4c78693567"
server_key_iv: "202adb4ec3851a2d269f472f53e7cbf7"
server_key_version: "encryptionkey@0"
I20260928 16:50:18.221273 15587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "fdad8e24cf234786b2edce754c78d9ef"
format_stamp: "Formatted at 2026-09-28 16:50:18 on dist-test-slave-wzcb"
server_key: "ee580012edb813ae8a929f4c78693567"
server_key_iv: "202adb4ec3851a2d269f472f53e7cbf7"
server_key_version: "encryptionkey@0"
I20260928 16:50:18.226871 15587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:50:18.230440 15446 heartbeater.cc:503] Master 127.9.93.62:33441 was elected leader, sending a full tablet report...
I20260928 16:50:18.230785 15602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:18.232522 15587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:50:18.232714 15587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "fdad8e24cf234786b2edce754c78d9ef"
format_stamp: "Formatted at 2026-09-28 16:50:18 on dist-test-slave-wzcb"
server_key: "ee580012edb813ae8a929f4c78693567"
server_key_iv: "202adb4ec3851a2d269f472f53e7cbf7"
server_key_version: "encryptionkey@0"
I20260928 16:50:18.232908 15587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:18.248842 15587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:18.253293 15587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:18.253563 15587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:18.254585 15587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:50:18.256070 15587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:18.256150 15587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:18.256247 15587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:18.256297 15587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:18.292235 15587 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:33349
I20260928 16:50:18.292299 15715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:33349 every 8 connection(s)
I20260928 16:50:18.293682 15587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:50:18.296037 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 15587
I20260928 16:50:18.296219 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:50:18.296787 9588 external_mini_cluster.cc:1428] Setting key c4722a38c7923984a0b8b56652431f4d
Sep 28 16:50:18 dist-test-slave-wzcb krb5kdc[15212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614218, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:50:18.314791 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:18.296502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:33237 (local address 127.9.93.62:33441)
0928 16:50:18.296990 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:18.296997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:18.298205 (+ 1208us) server_negotiation.cc:407] Connection header received
0928 16:50:18.300311 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:18.300315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:18.300385 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:18.300473 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:18.302068 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:18.302824 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:18.304432 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:18.304862 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:18.304957 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:18.308627 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:18.308632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:18.308636 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:18.308672 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:18.311605 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:18.312516 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:18.312523 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:18.312526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:18.312736 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:18.313227 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:18.313231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:18.313233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:18.313519 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:18.313633 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:18.314317 (+ 684us) server_negotiation.cc:299] Negotiation successful
0928 16:50:18.314497 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":226,"threads_started":1}
I20260928 16:50:18.315892 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:18.297147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33441 (local address 127.9.93.3:33237)
0928 16:50:18.298048 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:18.298089 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:18.300055 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:18.300712 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:18.300721 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:18.301273 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:18.301898 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:18.301918 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:18.302945 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:18.302948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:18.304293 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:18.304306 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:18.305072 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:18.305078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:18.305312 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:18.305986 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:18.306011 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:18.308400 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:18.311745 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:18.311755 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:18.311775 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:18.312357 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:18.312869 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:18.312875 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:18.312879 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:18.313102 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:18.313614 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:18.313620 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:18.313976 (+ 356us) client_negotiation.cc:769] Sending connection context
0928 16:50:18.314234 (+ 258us) client_negotiation.cc:240] Negotiation successful
0928 16:50:18.314661 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":375,"threads_started":1}
I20260928 16:50:18.317302 15716 heartbeater.cc:347] Connected to a master server at 127.9.93.62:33441
I20260928 16:50:18.317658 15716 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:18.318468 15716 heartbeater.cc:511] Master 127.9.93.62:33441 requested a full tablet report, sending...
I20260928 16:50:18.320124 15257 ts_manager.cc:194] Registered new tserver with Master: fdad8e24cf234786b2edce754c78d9ef (127.9.93.3:33349)
I20260928 16:50:18.320966 15257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.3@KRBTEST.COM'} at 127.9.93.3:33237
I20260928 16:50:18.324892 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/test-admin.keytab.
Sep 28 16:50:18 dist-test-slave-wzcb krb5kdc[15212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:18.353852 9588 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 16:50:18.359606 15724 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:18.356906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42626 (local address 127.9.93.1:37241)
0928 16:50:18.357679 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:18.357686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:18.357715 (+ 29us) server_negotiation.cc:407] Connection header received
0928 16:50:18.357783 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:18.357790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:18.358031 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:18.358167 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:18.359134 (+ 967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42626: BlockingRecv error: recv got EOF from 127.0.0.1:42626 (error 108)
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":418,"threads_started":1}
W20260928 16:50:18.360318 9588 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/krb5cc' of type FILE
Sep 28 16:50:18 dist-test-slave-wzcb krb5kdc[15212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:18.361991 9588 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 16:50:18 dist-test-slave-wzcb krb5kdc[15212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:18.363462 9588 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 16:50:18 dist-test-slave-wzcb krb5kdc[15212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.1@KRBTEST.COM
I20260928 16:50:18.379773 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:18.365396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42640 (local address 127.9.93.1:37241)
0928 16:50:18.365668 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:18.365675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:18.365807 (+ 132us) server_negotiation.cc:407] Connection header received
0928 16:50:18.366012 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:18.366016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:18.366089 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:18.366178 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:18.367245 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:18.368331 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:18.369577 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:18.370212 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:18.370337 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:18.374083 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:18.374097 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:18.374121 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:18.374162 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:18.376987 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:18.377650 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:18.377658 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:18.377660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:18.377795 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:18.378154 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:18.378157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:18.378160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:18.378546 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:18.378838 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:18.379180 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 16:50:18.379493 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
W20260928 16:50:18.386204 15724 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:18.384288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42648 (local address 127.9.93.1:37241)
0928 16:50:18.384497 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:18.384502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:18.384709 (+ 207us) server_negotiation.cc:407] Connection header received
0928 16:50:18.384840 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:18.384843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:18.384918 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:18.384996 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:18.385936 (+ 940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42648: BlockingRecv error: recv got EOF from 127.0.0.1:42648 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260928 16:50:18.386432 9588 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/krb5cc' of type FILE
Sep 28 16:50:18 dist-test-slave-wzcb krb5kdc[15212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:18.388047 9588 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 16:50:18 dist-test-slave-wzcb krb5kdc[15212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:18.389344 9588 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 16:50:18 dist-test-slave-wzcb krb5kdc[15212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.1@KRBTEST.COM
I20260928 16:50:18.403399 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:18.390730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42662 (local address 127.9.93.1:37241)
0928 16:50:18.390936 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:18.390943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:18.391191 (+ 248us) server_negotiation.cc:407] Connection header received
0928 16:50:18.391351 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:18.391355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:18.391423 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:18.391501 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:18.392510 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:18.393341 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:18.394589 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:18.395062 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:18.395158 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:18.398586 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:18.398592 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:18.398595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:18.398634 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:18.401115 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:18.401749 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:18.401753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:18.401755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:18.401896 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:18.402192 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:18.402195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:18.402197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:18.402470 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:18.402689 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:18.402862 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 16:50:18.403051 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 16:50:18.407907 15724 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:18.405980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42670 (local address 127.9.93.1:37241)
0928 16:50:18.406186 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:18.406192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:18.406447 (+ 255us) server_negotiation.cc:407] Connection header received
0928 16:50:18.406602 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:18.406615 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:18.406699 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:18.406781 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:18.407641 (+ 860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42670: BlockingRecv error: recv got EOF from 127.0.0.1:42670 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 16:50:18.408010 9588 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790614166338986-9588-0/krb5kdc/krb5cc' of type FILE
Sep 28 16:50:18 dist-test-slave-wzcb krb5kdc[15212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:18.409550 9588 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 16:50:18 dist-test-slave-wzcb krb5kdc[15212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:18.410786 9588 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 16:50:18 dist-test-slave-wzcb krb5kdc[15212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.1@KRBTEST.COM
I20260928 16:50:18.425814 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:18.412344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42674 (local address 127.9.93.1:37241)
0928 16:50:18.412573 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:18.412580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:18.412782 (+ 202us) server_negotiation.cc:407] Connection header received
0928 16:50:18.412922 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:18.412928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:18.413004 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:18.413088 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:18.414111 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:18.414868 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:18.416158 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:18.416615 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:18.416729 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:18.420360 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:18.420366 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:18.420369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:18.420408 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:18.423013 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:18.423767 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:18.423773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:18.423777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:18.423954 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:18.424318 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:18.424323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:18.424325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:18.424649 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:18.424977 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:18.425241 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 16:50:18.425482 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 16:50:18.427661 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 15318
I20260928 16:50:18.439713 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:18.428369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42688 (local address 127.9.93.1:37241)
0928 16:50:18.428523 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:18.428528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:18.428890 (+ 362us) server_negotiation.cc:407] Connection header received
0928 16:50:18.429016 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:18.429020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:18.429078 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:18.429156 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:18.430043 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:18.430750 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:18.431945 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:18.432473 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:18.432565 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:18.434632 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:18.434637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:18.434640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:18.434674 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:18.437282 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:18.437904 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:18.437908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:18.437910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:18.438048 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:18.438381 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:18.438384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:18.438386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:18.438679 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:18.438896 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:18.439105 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 16:50:18.439294 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:50:18.440294 15441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:18.528059 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 15318
I20260928 16:50:18.545674 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 15452
Sep 28 16:50:18 dist-test-slave-wzcb krb5kdc[15212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.2@KRBTEST.COM
I20260928 16:50:18.563107 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:18.546782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45356 (local address 127.9.93.2:45373)
0928 16:50:18.547550 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:18.547558 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:18.547588 (+ 30us) server_negotiation.cc:407] Connection header received
0928 16:50:18.547667 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:18.547674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:18.547918 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:18.548108 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:18.549248 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:18.550382 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:18.551818 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:18.552456 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:18.552586 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:18.556758 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:18.556775 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:18.556793 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:18.556844 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:18.559842 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:18.560668 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:18.560678 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:18.560681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:18.560844 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:18.561291 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:18.561295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:18.561297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:18.561721 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:18.562061 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:18.562434 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 16:50:18.562754 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":394,"threads_started":1}
I20260928 16:50:18.564574 15575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:18.653306 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 15452
I20260928 16:50:18.667950 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 15587
Sep 28 16:50:18 dist-test-slave-wzcb krb5kdc[15212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.3@KRBTEST.COM
I20260928 16:50:18.695533 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:18.669113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41564 (local address 127.9.93.3:33349)
0928 16:50:18.669781 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:18.669789 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:18.669823 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:50:18.669893 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:18.669900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:18.670147 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:18.670295 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:18.671524 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:18.672713 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:18.674102 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:18.674771 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:18.674893 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:18.688668 (+ 13775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:18.688690 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:18.688709 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:18.688765 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:18.691899 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:18.692862 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:18.692872 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:18.692874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:18.693035 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:18.693370 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:18.693373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:18.693375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:18.693793 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:18.694071 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:18.694677 (+ 606us) server_negotiation.cc:299] Negotiation successful
0928 16:50:18.695057 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":303,"threads_started":1}
I20260928 16:50:18.697280 15711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:18.786008 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 15587
I20260928 16:50:18.799973 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 15225
I20260928 16:50:18.801229 15287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:18.906716 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 15225
2026-09-28T16:50:18Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4066 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:50:19 dist-test-slave-wzcb krb5kdc[15744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:50:19 dist-test-slave-wzcb krb5kdc[15744](info): set up 2 sockets
Sep 28 16:50:19 dist-test-slave-wzcb krb5kdc[15744](info): commencing operation
krb5kdc: starting...
W20260928 16:50:19.926545 9588 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:50:20 dist-test-slave-wzcb krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:50:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:50:20Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.62.keytab.
Entry for principal oryx/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.62.keytab.
I20260928 16:50:20.167824 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:33733
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.62.keytab
--principal=oryx/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:39869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:33733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:50:20.522609 15757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:20.522956 15757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:20.523092 15757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:20.530789 15757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:50:20.530922 15757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:20.530983 15757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:20.531064 15757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:50:20.531126 15757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:50:20.542539 15757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:33733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.62.keytab
--principal=oryx/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:33733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:20.544181 15757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:20.545717 15757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:20.558444 15762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:20.558393 15763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:20.558415 15765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:20.559988 15757 server_base.cc:1034] running on GCE node
I20260928 16:50:20.560982 15757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:20.562734 15757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:20.563967 15757 hybrid_clock.cc:648] HybridClock initialized: now 1790614220563913 us; error 46 us; skew 500 ppm
I20260928 16:50:20.564360 15757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:20 dist-test-slave-wzcb krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614220, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:20.567787 15757 init.cc:377] Logged in from keytab as oryx/127.9.93.62@KRBTEST.COM (short username oryx)
I20260928 16:50:20.569483 15757 webserver.cc:460] Webserver started at http://127.9.93.62:44609/ using document root <none> and password file <none>
I20260928 16:50:20.570292 15757 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:20.570413 15757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:20.570747 15757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:20.574151 15757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "2577aff545514d4389450abad8337d7b"
format_stamp: "Formatted at 2026-09-28 16:50:20 on dist-test-slave-wzcb"
server_key: "c8dbc27d410662e7c73ba96fe1b0c0af"
server_key_iv: "92a1f2c5927f68e61542b9b0105a47e5"
server_key_version: "encryptionkey@0"
I20260928 16:50:20.575012 15757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "2577aff545514d4389450abad8337d7b"
format_stamp: "Formatted at 2026-09-28 16:50:20 on dist-test-slave-wzcb"
server_key: "c8dbc27d410662e7c73ba96fe1b0c0af"
server_key_iv: "92a1f2c5927f68e61542b9b0105a47e5"
server_key_version: "encryptionkey@0"
I20260928 16:50:20.580991 15757 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260928 16:50:20.585103 15772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:20.586758 15757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:50:20.586946 15757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "2577aff545514d4389450abad8337d7b"
format_stamp: "Formatted at 2026-09-28 16:50:20 on dist-test-slave-wzcb"
server_key: "c8dbc27d410662e7c73ba96fe1b0c0af"
server_key_iv: "92a1f2c5927f68e61542b9b0105a47e5"
server_key_version: "encryptionkey@0"
I20260928 16:50:20.587155 15757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:20.622079 15757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:20.627032 15757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:20.627368 15757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:20.647938 15757 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:33733
I20260928 16:50:20.647976 15824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:33733 every 8 connection(s)
I20260928 16:50:20.649520 15757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:50:20.652231 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 15757
I20260928 16:50:20.652439 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:50:20.653203 9588 external_mini_cluster.cc:1428] Setting key e2f1e8576b2c48cded118345cb9aea85
I20260928 16:50:20.654125 15825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 16:50:20 dist-test-slave-wzcb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM
I20260928 16:50:20.670131 15825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b: Bootstrap starting.
I20260928 16:50:20.674872 15825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:20.676422 15825 log.cc:824] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b: Log is configured to *not* fsync() on all Append() calls
I20260928 16:50:20.680589 15825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b: No bootstrap required, opened a new log
I20260928 16:50:20.680619 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:20.655604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38340 (local address 127.9.93.62:33733)
0928 16:50:20.656521 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:20.656540 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:20.656598 (+ 58us) server_negotiation.cc:407] Connection header received
0928 16:50:20.658355 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:20.658394 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:20.659092 (+ 698us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:20.659684 (+ 592us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:20.661186 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:20.663442 (+ 2256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:20.664817 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:20.665933 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:20.666166 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:20.670915 (+ 4749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:20.670927 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:20.670951 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:20.671024 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:20.675174 (+ 4150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:20.676120 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:20.676131 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:20.676142 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:20.676383 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:20.676870 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:20.676875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:20.676878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:20.677667 (+ 789us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:20.677937 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:20.678575 (+ 638us) server_negotiation.cc:299] Negotiation successful
0928 16:50:20.679107 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":369,"threads_started":1}
I20260928 16:50:20.689489 15825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } }
I20260928 16:50:20.689962 15825 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:20.690083 15825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2577aff545514d4389450abad8337d7b, State: Initialized, Role: FOLLOWER
I20260928 16:50:20.690743 15825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } }
I20260928 16:50:20.691008 15825 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:50:20.691152 15825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:50:20.691403 15825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:20.694123 15825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } }
I20260928 16:50:20.694859 15825 leader_election.cc:302] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2577aff545514d4389450abad8337d7b; no voters:
I20260928 16:50:20.695590 15825 leader_election.cc:288] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:50:20.695715 15830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:50:20.696415 15830 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [term 1 LEADER]: Becoming Leader. State: Replica: 2577aff545514d4389450abad8337d7b, State: Running, Role: LEADER
I20260928 16:50:20.697007 15830 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } }
I20260928 16:50:20.697974 15825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:50:20.699896 15831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2577aff545514d4389450abad8337d7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } }
I20260928 16:50:20.700430 15831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:20.700228 15832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2577aff545514d4389450abad8337d7b. Latest consensus state: current_term: 1 leader_uuid: "2577aff545514d4389450abad8337d7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } }
I20260928 16:50:20.700584 15832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:20.701505 15836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:50:20.712831 15836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:50:20.728119 15836 catalog_manager.cc:1347] Generated new cluster ID: 5bcd09d1c1b14d24a6129be96c99a994
I20260928 16:50:20.728273 15836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:50:20.767489 15836 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:50:20.769671 15836 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:50:20.783106 15836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b: Generated new TSK 0
I20260928 16:50:20.784479 15836 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.1.keytab.
Entry for principal oryx/127.9.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.1.keytab.
I20260928 16:50:20.847404 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33733
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.1.keytab
--principal=oryx/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:39869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:50:21.233652 15851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:21.233994 15851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:21.234153 15851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:21.242357 15851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:21.242503 15851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:21.242679 15851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:50:21.253695 15851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.1.keytab
--principal=oryx/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:21.255268 15851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:21.256915 15851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:21.272222 15857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:21.272361 15856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:21.272353 15859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:21.273401 15851 server_base.cc:1034] running on GCE node
I20260928 16:50:21.274120 15851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:21.275245 15851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:21.276533 15851 hybrid_clock.cc:648] HybridClock initialized: now 1790614221276459 us; error 53 us; skew 500 ppm
I20260928 16:50:21.276935 15851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:21 dist-test-slave-wzcb krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614221, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:21.280536 15851 init.cc:377] Logged in from keytab as oryx/127.9.93.1@KRBTEST.COM (short username oryx)
I20260928 16:50:21.282318 15851 webserver.cc:460] Webserver started at http://127.9.93.1:41979/ using document root <none> and password file <none>
I20260928 16:50:21.283195 15851 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:21.283360 15851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:21.283712 15851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:21.287187 15851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "8c5622e16bb5462793db42b715d8d31b"
format_stamp: "Formatted at 2026-09-28 16:50:21 on dist-test-slave-wzcb"
server_key: "8b9bf11015244de2cfef9ae45ac71c89"
server_key_iv: "a87ddf20c84ce13dde202bc96dc44168"
server_key_version: "encryptionkey@0"
I20260928 16:50:21.288188 15851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "8c5622e16bb5462793db42b715d8d31b"
format_stamp: "Formatted at 2026-09-28 16:50:21 on dist-test-slave-wzcb"
server_key: "8b9bf11015244de2cfef9ae45ac71c89"
server_key_iv: "a87ddf20c84ce13dde202bc96dc44168"
server_key_version: "encryptionkey@0"
I20260928 16:50:21.294545 15851 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 16:50:21.299062 15866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:21.300833 15851 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:50:21.301051 15851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "8c5622e16bb5462793db42b715d8d31b"
format_stamp: "Formatted at 2026-09-28 16:50:21 on dist-test-slave-wzcb"
server_key: "8b9bf11015244de2cfef9ae45ac71c89"
server_key_iv: "a87ddf20c84ce13dde202bc96dc44168"
server_key_version: "encryptionkey@0"
I20260928 16:50:21.301299 15851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:21.324988 15851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:21.330291 15851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:21.330619 15851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:21.331769 15851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:50:21.333346 15851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:21.333433 15851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:21.333599 15851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:21.333675 15851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:21.366699 15851 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:40577
I20260928 16:50:21.366751 15979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:40577 every 8 connection(s)
I20260928 16:50:21.368254 15851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:50:21.375034 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 15851
I20260928 16:50:21.375288 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:50:21.376163 9588 external_mini_cluster.cc:1428] Setting key a1b1db3a3f0e67c8e5c5b0ce70ed36a3
Sep 28 16:50:21 dist-test-slave-wzcb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614221, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM
I20260928 16:50:21.392287 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:21.371224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:43691 (local address 127.9.93.62:33733)
0928 16:50:21.371746 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:21.371752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:21.373024 (+ 1272us) server_negotiation.cc:407] Connection header received
0928 16:50:21.375431 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:21.375439 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:21.375526 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:21.375663 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:21.377869 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:21.378633 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:21.380402 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:21.380825 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:21.380943 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:21.385057 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:21.385063 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:21.385067 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:21.385106 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:21.388360 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:21.389377 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:21.389383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:21.389386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:21.389543 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:21.390117 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:21.390121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:21.390124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:21.390508 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:21.390670 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:21.391648 (+ 978us) server_negotiation.cc:299] Negotiation successful
0928 16:50:21.391910 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":248,"threads_started":1}
I20260928 16:50:21.393523 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:21.371970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.1:43691)
0928 16:50:21.372839 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:21.372884 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:21.374979 (+ 2095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:21.375998 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:21.376013 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:21.376860 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:21.377679 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:21.377701 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:21.378804 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:21.378808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:21.380232 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:21.380247 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:21.381077 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:21.381094 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:21.381340 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:21.382045 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:21.382071 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:21.384767 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:21.388593 (+ 3826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:21.388602 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:21.388623 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:21.389211 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:21.389701 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:21.389706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:21.389708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:21.389985 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:21.390665 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:21.390674 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:21.391088 (+ 414us) client_negotiation.cc:769] Sending connection context
0928 16:50:21.391490 (+ 402us) client_negotiation.cc:240] Negotiation successful
0928 16:50:21.392059 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":320,"threads_started":1}
I20260928 16:50:21.395584 15980 heartbeater.cc:347] Connected to a master server at 127.9.93.62:33733
I20260928 16:50:21.395941 15980 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:21.396795 15980 heartbeater.cc:511] Master 127.9.93.62:33733 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.9.93.2@KRBTEST.COM; defaulting to no policy
I20260928 16:50:21.399596 15789 ts_manager.cc:194] Registered new tserver with Master: 8c5622e16bb5462793db42b715d8d31b (127.9.93.1:40577)
I20260928 16:50:21.401278 15789 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.93.1@KRBTEST.COM'} at 127.9.93.1:43691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.2.keytab.
Entry for principal oryx/127.9.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.2.keytab.
I20260928 16:50:21.430792 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33733
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.2.keytab
--principal=oryx/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:39869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:50:21.806403 15986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:21.806746 15986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:21.806857 15986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:21.814821 15986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:21.814956 15986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:21.815092 15986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:50:21.826117 15986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.2.keytab
--principal=oryx/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:21.827740 15986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:21.829281 15986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:21.847990 15994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:21.848037 15992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:21.847959 15991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:21.849352 15986 server_base.cc:1034] running on GCE node
I20260928 16:50:21.850348 15986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:21.851584 15986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:21.852818 15986 hybrid_clock.cc:648] HybridClock initialized: now 1790614221852756 us; error 56 us; skew 500 ppm
I20260928 16:50:21.853202 15986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:21 dist-test-slave-wzcb krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614221, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:21.856845 15986 init.cc:377] Logged in from keytab as oryx/127.9.93.2@KRBTEST.COM (short username oryx)
I20260928 16:50:21.858599 15986 webserver.cc:460] Webserver started at http://127.9.93.2:43309/ using document root <none> and password file <none>
I20260928 16:50:21.859484 15986 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:21.859622 15986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:21.859956 15986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:21.863467 15986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "d29db76b68474ab6aa678633dd238fe1"
format_stamp: "Formatted at 2026-09-28 16:50:21 on dist-test-slave-wzcb"
server_key: "990362f1b065631a66346e900fedee7e"
server_key_iv: "5342072b0630725b4f35b03e661039ca"
server_key_version: "encryptionkey@0"
I20260928 16:50:21.864394 15986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "d29db76b68474ab6aa678633dd238fe1"
format_stamp: "Formatted at 2026-09-28 16:50:21 on dist-test-slave-wzcb"
server_key: "990362f1b065631a66346e900fedee7e"
server_key_iv: "5342072b0630725b4f35b03e661039ca"
server_key_version: "encryptionkey@0"
I20260928 16:50:21.870694 15986 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 16:50:21.875113 16001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:21.876858 15986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:50:21.877085 15986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "d29db76b68474ab6aa678633dd238fe1"
format_stamp: "Formatted at 2026-09-28 16:50:21 on dist-test-slave-wzcb"
server_key: "990362f1b065631a66346e900fedee7e"
server_key_iv: "5342072b0630725b4f35b03e661039ca"
server_key_version: "encryptionkey@0"
I20260928 16:50:21.877349 15986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:21.890848 15986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:21.895798 15986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:21.896129 15986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:21.897224 15986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:50:21.898921 15986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:21.899009 15986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:21.899155 15986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:21.899216 15986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:21.932777 15986 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:43275
I20260928 16:50:21.932823 16114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:43275 every 8 connection(s)
I20260928 16:50:21.934257 15986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:50:21.935877 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 15986
I20260928 16:50:21.936072 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:50:21.936555 9588 external_mini_cluster.cc:1428] Setting key b32948db9a4f49304c1e44ba25c7c454
Sep 28 16:50:21 dist-test-slave-wzcb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614221, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM
I20260928 16:50:21.959126 16117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:21.937205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:42851 (local address 127.9.93.62:33733)
0928 16:50:21.937776 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:21.937780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:21.939035 (+ 1255us) server_negotiation.cc:407] Connection header received
0928 16:50:21.941877 (+ 2842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:21.941882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:21.941963 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:21.942067 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:21.944538 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:21.945364 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:21.947225 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:21.947770 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:21.947904 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:21.952409 (+ 4505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:21.952415 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:21.952419 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:21.952460 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:21.955177 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:21.956203 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:21.956209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:21.956212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:21.956370 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:21.956929 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:21.956934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:21.956936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:21.957270 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:21.957427 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:21.958329 (+ 902us) server_negotiation.cc:299] Negotiation successful
0928 16:50:21.958615 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":265,"threads_started":1}
I20260928 16:50:21.960307 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:21.937925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.2:42851)
0928 16:50:21.938835 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:21.938887 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:21.941511 (+ 2624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:21.942405 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:21.942418 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:21.943281 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:21.944298 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:21.944329 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:21.945555 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:21.945559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:21.947018 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:21.947037 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:21.948081 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:21.948090 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:21.948485 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:21.949391 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:21.949428 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:21.952081 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:21.955433 (+ 3352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:21.955444 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:21.955462 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:21.956031 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:21.956510 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:21.956516 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:21.956519 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:21.956784 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:21.957427 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:21.957438 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:21.957897 (+ 459us) client_negotiation.cc:769] Sending connection context
0928 16:50:21.958190 (+ 293us) client_negotiation.cc:240] Negotiation successful
0928 16:50:21.958920 (+ 730us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":353,"threads_started":1}
WARNING: no policy specified for oryx/127.9.93.3@KRBTEST.COM; defaulting to no policy
I20260928 16:50:21.962051 16115 heartbeater.cc:347] Connected to a master server at 127.9.93.62:33733
I20260928 16:50:21.962402 16115 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:21.963291 16115 heartbeater.cc:511] Master 127.9.93.62:33733 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.93.3@KRBTEST.COM" created.
I20260928 16:50:21.965158 15789 ts_manager.cc:194] Registered new tserver with Master: d29db76b68474ab6aa678633dd238fe1 (127.9.93.2:43275)
I20260928 16:50:21.966321 15789 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.93.2@KRBTEST.COM'} at 127.9.93.2:42851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.3.keytab.
Entry for principal oryx/127.9.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.3.keytab.
I20260928 16:50:21.995137 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33733
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.3.keytab
--principal=oryx/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:39869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:50:22.378700 16121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:22.379094 16121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:22.379230 16121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:22.387357 16121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:22.387509 16121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:22.387657 16121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:50:22.398725 16121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.3.keytab
--principal=oryx/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:22.400379 16121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:22.401877 16121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:50:22.404919 15980 heartbeater.cc:503] Master 127.9.93.62:33733 was elected leader, sending a full tablet report...
W20260928 16:50:22.417266 16126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:22.417243 16129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:22.417300 16127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:22.418092 16121 server_base.cc:1034] running on GCE node
I20260928 16:50:22.418835 16121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:22.420106 16121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:22.421360 16121 hybrid_clock.cc:648] HybridClock initialized: now 1790614222421301 us; error 53 us; skew 500 ppm
I20260928 16:50:22.421727 16121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:22 dist-test-slave-wzcb krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614222, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:22.425256 16121 init.cc:377] Logged in from keytab as oryx/127.9.93.3@KRBTEST.COM (short username oryx)
I20260928 16:50:22.426914 16121 webserver.cc:460] Webserver started at http://127.9.93.3:43127/ using document root <none> and password file <none>
I20260928 16:50:22.427807 16121 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:22.427925 16121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:22.428227 16121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:22.431654 16121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "7397325a370249b5a66a0177c1e2fa13"
format_stamp: "Formatted at 2026-09-28 16:50:22 on dist-test-slave-wzcb"
server_key: "6d1e3bafcfc0b7d553b6b90021fbd017"
server_key_iv: "3b362487759afebc6f0490600a023fe6"
server_key_version: "encryptionkey@0"
I20260928 16:50:22.432533 16121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "7397325a370249b5a66a0177c1e2fa13"
format_stamp: "Formatted at 2026-09-28 16:50:22 on dist-test-slave-wzcb"
server_key: "6d1e3bafcfc0b7d553b6b90021fbd017"
server_key_iv: "3b362487759afebc6f0490600a023fe6"
server_key_version: "encryptionkey@0"
I20260928 16:50:22.438827 16121 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 16:50:22.443096 16136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:22.444859 16121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:50:22.445046 16121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "7397325a370249b5a66a0177c1e2fa13"
format_stamp: "Formatted at 2026-09-28 16:50:22 on dist-test-slave-wzcb"
server_key: "6d1e3bafcfc0b7d553b6b90021fbd017"
server_key_iv: "3b362487759afebc6f0490600a023fe6"
server_key_version: "encryptionkey@0"
I20260928 16:50:22.445266 16121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:22.472735 16121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:22.477265 16121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:22.477552 16121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:22.478626 16121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:50:22.480218 16121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:22.480301 16121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:22.480403 16121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:22.480448 16121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:22.513867 16121 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:42893
I20260928 16:50:22.513916 16249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:42893 every 8 connection(s)
I20260928 16:50:22.515357 16121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:50:22.521996 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 16121
I20260928 16:50:22.522207 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:50:22.522789 9588 external_mini_cluster.cc:1428] Setting key 47341185e5ea9dff799c932a0bd1fa3d
Sep 28 16:50:22 dist-test-slave-wzcb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614222, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM
I20260928 16:50:22.537627 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:22.518230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:37495 (local address 127.9.93.62:33733)
0928 16:50:22.518746 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:22.518751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:22.519885 (+ 1134us) server_negotiation.cc:407] Connection header received
0928 16:50:22.521967 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:22.521973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:22.522085 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:22.522203 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:22.524369 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:22.525395 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:22.527368 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:22.527808 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:22.527911 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:22.531708 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:22.531713 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:22.531717 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:22.531755 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:22.534279 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:22.535083 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:22.535087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:22.535090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:22.535214 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:22.535662 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:22.535665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:22.535667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:22.535957 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:22.536078 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:22.536999 (+ 921us) server_negotiation.cc:299] Negotiation successful
0928 16:50:22.537238 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":234,"threads_started":1}
I20260928 16:50:22.538909 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:22.518848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.3:37495)
0928 16:50:22.519716 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:22.519759 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:22.521650 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:22.522550 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:22.522565 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:22.523450 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:22.524188 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:22.524211 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:22.525576 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:22.525580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:22.527171 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:22.527193 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:22.528012 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:22.528017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:22.528267 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:22.528967 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:22.528993 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:22.531421 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:22.534438 (+ 3017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:22.534447 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:22.534466 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:22.534974 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:22.535292 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:22.535295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:22.535298 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:22.535544 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:22.536059 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:22.536065 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:22.536484 (+ 419us) client_negotiation.cc:769] Sending connection context
0928 16:50:22.536844 (+ 360us) client_negotiation.cc:240] Negotiation successful
0928 16:50:22.537428 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":290,"threads_started":1}
I20260928 16:50:22.540412 16250 heartbeater.cc:347] Connected to a master server at 127.9.93.62:33733
I20260928 16:50:22.540761 16250 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:22.541567 16250 heartbeater.cc:511] Master 127.9.93.62:33733 requested a full tablet report, sending...
I20260928 16:50:22.543193 15789 ts_manager.cc:194] Registered new tserver with Master: 7397325a370249b5a66a0177c1e2fa13 (127.9.93.3:42893)
I20260928 16:50:22.544100 15789 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.93.3@KRBTEST.COM'} at 127.9.93.3:37495
I20260928 16:50:22.551738 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.93.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.93.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.93.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.61.keytab.
Entry for principal oryx/127.9.93.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.61.keytab.
I20260928 16:50:22.603801 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.61:38393
--webserver_interface=127.9.93.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.61.keytab
--principal=oryx/127.9.93.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:39869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:33733,127.9.93.61:38393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:50:22.969527 16115 heartbeater.cc:503] Master 127.9.93.62:33733 was elected leader, sending a full tablet report...
W20260928 16:50:22.986624 16256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:22.986974 16256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:22.987084 16256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:22.995189 16256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:50:22.995388 16256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:22.995500 16256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:22.995569 16256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:50:22.995620 16256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:50:23.007297 16256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:33733,127.9.93.61:38393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.61.keytab
--principal=oryx/127.9.93.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.61:38393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.93.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
W20260928 16:50:23.007745 16256 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.9.93.62:33733,127.9.93.61:38393'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 16:50:23.008993 16256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:23.010576 16256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:23.025341 16265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:23.025319 16262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:23.025527 16261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:23.026310 16256 server_base.cc:1034] running on GCE node
I20260928 16:50:23.027372 16256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:23.029173 16256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:23.030452 16256 hybrid_clock.cc:648] HybridClock initialized: now 1790614223030337 us; error 76 us; skew 500 ppm
I20260928 16:50:23.030874 16256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:23 dist-test-slave-wzcb krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614223, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:23.034529 16256 init.cc:377] Logged in from keytab as oryx/127.9.93.61@KRBTEST.COM (short username oryx)
I20260928 16:50:23.036382 16256 webserver.cc:460] Webserver started at http://127.9.93.61:37325/ using document root <none> and password file <none>
I20260928 16:50:23.037212 16256 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:23.037324 16256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:23.037691 16256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:23.041452 16256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/data/instance:
uuid: "31a246c9c6804fdb8430171f57199a93"
format_stamp: "Formatted at 2026-09-28 16:50:23 on dist-test-slave-wzcb"
server_key: "b25b70975d4abfef847f675aa3faef28"
server_key_iv: "227dc9c8d924d57f1b5b7fd84925e36e"
server_key_version: "encryptionkey@0"
I20260928 16:50:23.042402 16256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/wal/instance:
uuid: "31a246c9c6804fdb8430171f57199a93"
format_stamp: "Formatted at 2026-09-28 16:50:23 on dist-test-slave-wzcb"
server_key: "b25b70975d4abfef847f675aa3faef28"
server_key_iv: "227dc9c8d924d57f1b5b7fd84925e36e"
server_key_version: "encryptionkey@0"
I20260928 16:50:23.048918 16256 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 16:50:23.053285 16272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:23.054911 16256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:50:23.055112 16256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/wal
uuid: "31a246c9c6804fdb8430171f57199a93"
format_stamp: "Formatted at 2026-09-28 16:50:23 on dist-test-slave-wzcb"
server_key: "b25b70975d4abfef847f675aa3faef28"
server_key_iv: "227dc9c8d924d57f1b5b7fd84925e36e"
server_key_version: "encryptionkey@0"
I20260928 16:50:23.055450 16256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:23.078601 16256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:23.083941 16256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:23.084245 16256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:23.104903 16256 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.61:38393
I20260928 16:50:23.104923 16323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.61:38393 every 8 connection(s)
I20260928 16:50:23.106580 16256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/data/info.pb
I20260928 16:50:23.108750 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 16256
I20260928 16:50:23.112079 16324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:23.122028 16324 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:50:23 dist-test-slave-wzcb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.93.61@KRBTEST.COM
Sep 28 16:50:23 dist-test-slave-wzcb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614223, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.61@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM
I20260928 16:50:23.137517 16327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.111488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52394 (local address 127.9.93.61:38393)
0928 16:50:23.112454 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:23.112469 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:23.112520 (+ 51us) server_negotiation.cc:407] Connection header received
0928 16:50:23.114378 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:23.114407 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:23.114919 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:23.115449 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:23.117562 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.119128 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.121132 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.122047 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.122231 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:23.127777 (+ 5546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:23.127788 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:23.127810 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:23.127865 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:23.131859 (+ 3994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:23.132781 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:23.132789 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:23.132798 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:23.133011 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:23.133518 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:23.133523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:23.133526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:23.134382 (+ 856us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:23.134753 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:23.135374 (+ 621us) server_negotiation.cc:299] Negotiation successful
0928 16:50:23.135921 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":439,"threads_started":1}
I20260928 16:50:23.143467 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.126019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32804 (local address 127.9.93.62:33733)
0928 16:50:23.126539 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:23.126555 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:23.127446 (+ 891us) server_negotiation.cc:407] Connection header received
0928 16:50:23.127672 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:23.127677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:23.127787 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:23.127907 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:23.129529 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.130724 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.132437 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.133001 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.133137 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:23.137199 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:23.137205 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:23.137209 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:23.137259 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:23.140192 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:23.141171 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:23.141176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:23.141179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:23.141321 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:23.141722 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:23.141726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:23.141728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:23.142105 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:23.142255 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:23.142763 (+ 508us) server_negotiation.cc:299] Negotiation successful
0928 16:50:23.143046 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":262,"threads_started":1}
I20260928 16:50:23.143515 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.126045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.0.0.1:32804)
0928 16:50:23.127239 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:23.127327 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:23.127526 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:23.127954 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:23.127960 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:23.128954 (+ 994us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:23.129322 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:23.129337 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.130895 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.130900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:23.132267 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:23.132282 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.133211 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.133217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:23.133369 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:23.134019 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:23.134050 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:23.136986 (+ 2936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:23.140355 (+ 3369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:23.140370 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:23.140378 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:23.141019 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:23.141437 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:23.141442 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:23.141446 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:23.141606 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:23.142248 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:23.142257 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:23.142448 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 16:50:23.142630 (+ 182us) client_negotiation.cc:240] Negotiation successful
0928 16:50:23.143057 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"thread_start_us":725,"threads_started":1}
I20260928 16:50:23.145507 16324 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:50:23 dist-test-slave-wzcb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614223, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.61@KRBTEST.COM for oryx/127.9.93.61@KRBTEST.COM
I20260928 16:50:23.159086 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.146415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:38393 (local address 127.0.0.1:52396)
0928 16:50:23.146618 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:23.146637 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:23.146742 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:23.146981 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:23.146984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:23.147427 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:23.147583 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:23.147592 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.148586 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.148590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:23.149570 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:23.149584 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.150404 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.150409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:23.150579 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:23.151138 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:23.151157 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:23.153475 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:23.156650 (+ 3175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:23.156655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:23.156658 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:23.157074 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:23.157466 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:23.157469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:23.157472 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:23.157635 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:23.158176 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:23.158181 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:23.158343 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 16:50:23.158475 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 16:50:23.158798 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 16:50:23.159343 16327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.146399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52396 (local address 127.9.93.61:38393)
0928 16:50:23.146628 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:23.146633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:23.146693 (+ 60us) server_negotiation.cc:407] Connection header received
0928 16:50:23.146820 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:23.146824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:23.146884 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:23.146959 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:23.147717 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.148462 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.149701 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.150228 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.150328 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:23.153738 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:23.153744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:23.153747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:23.153785 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:23.156491 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:23.157193 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:23.157197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:23.157199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:23.157359 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:23.157723 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:23.157726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:23.157728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:23.158036 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:23.158162 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:23.158573 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 16:50:23.158955 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":58}
I20260928 16:50:23.165071 16324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93: Bootstrap starting.
I20260928 16:50:23.168874 16324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:23.170317 16324 log.cc:824] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93: Log is configured to *not* fsync() on all Append() calls
I20260928 16:50:23.173686 16324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93: No bootstrap required, opened a new log
I20260928 16:50:23.181999 16324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } }
I20260928 16:50:23.182415 16324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:23.182508 16324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a246c9c6804fdb8430171f57199a93, State: Initialized, Role: FOLLOWER
I20260928 16:50:23.183220 16324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } }
I20260928 16:50:23.184247 16331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } } }
I20260928 16:50:23.184607 16331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:50:23.185472 16324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 16:50:23.196237 16342 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:50:23.196430 16342 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:50:23.205439 16256 master_runner.cc:428] Detected that this master 31a246c9c6804fdb8430171f57199a93 is joining an existing cluster
I20260928 16:50:23.205608 16256 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.93.61:38393
WARNING: no policy specified for oryx/127.9.93.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.93.60@KRBTEST.COM" created.
I20260928 16:50:23.238457 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.215582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32816 (local address 127.9.93.62:33733)
0928 16:50:23.215804 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:23.215813 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:23.223031 (+ 7218us) server_negotiation.cc:407] Connection header received
0928 16:50:23.223187 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:23.223192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:23.223268 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:23.223392 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:23.224690 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.225758 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.227945 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.228583 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.228702 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:23.231700 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:23.231707 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:23.231711 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:23.231753 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:23.235124 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:23.235942 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:23.235948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:23.235952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:23.236118 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:23.236587 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:23.236592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:23.236595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:23.237004 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:23.237142 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:23.237797 (+ 655us) server_negotiation.cc:299] Negotiation successful
0928 16:50:23.238090 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 16:50:23.238848 16327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.215582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52406 (local address 127.9.93.61:38393)
0928 16:50:23.215835 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:23.215842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:23.222287 (+ 6445us) server_negotiation.cc:407] Connection header received
0928 16:50:23.222632 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:23.222638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:23.222731 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:23.222802 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:23.225070 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.226194 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.228332 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.229088 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.229200 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:23.231869 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:23.231875 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:23.231879 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:23.231910 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:23.235631 (+ 3721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:23.236573 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:23.236578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:23.236580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:23.236747 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:23.237166 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:23.237170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:23.237173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:23.237569 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:23.237713 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:23.238199 (+ 486us) server_negotiation.cc:299] Negotiation successful
0928 16:50:23.238452 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":60}
I20260928 16:50:23.239809 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.216083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.0.0.1:32816)
0928 16:50:23.222890 (+ 6807us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:23.222908 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:23.223060 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:23.223494 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:23.223500 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:23.224112 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:23.224460 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:23.224475 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.225995 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.226001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:23.227767 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:23.227783 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.229043 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.229049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:23.229260 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:23.229965 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:23.229988 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:23.231437 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:23.235343 (+ 3906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:23.235349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:23.235353 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:23.235781 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:23.236279 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:23.236284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:23.236288 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:23.236456 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:23.237219 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:23.237223 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:23.237459 (+ 236us) client_negotiation.cc:769] Sending connection context
0928 16:50:23.238765 (+ 1306us) client_negotiation.cc:240] Negotiation successful
0928 16:50:23.239275 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6666,"mutex_wait_us":43,"thread_start_us":6419,"threads_started":1}
I20260928 16:50:23.240660 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.215944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:38393 (local address 127.0.0.1:52406)
0928 16:50:23.222086 (+ 6142us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:23.222121 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:23.222457 (+ 336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:23.223023 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:23.223027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:23.224014 (+ 987us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:23.224899 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:23.224913 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.226440 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.226446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:23.228171 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:23.228191 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.229639 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.229647 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:23.229842 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:23.230443 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:23.230471 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:23.231684 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:23.235781 (+ 4097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:23.235787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:23.235790 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:23.236420 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:23.236887 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:23.236891 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:23.236894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:23.237067 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:23.237781 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:23.237785 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:23.237934 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 16:50:23.239494 (+ 1560us) client_negotiation.cc:240] Negotiation successful
0928 16:50:23.240206 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5923,"mutex_wait_us":26,"spinlock_wait_cycles":37632,"thread_start_us":5731,"threads_started":1}
Sep 28 16:50:23 dist-test-slave-wzcb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614220, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.62@KRBTEST.COM for oryx/127.9.93.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.93.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.60.keytab.
Entry for principal oryx/127.9.93.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.60.keytab.
I20260928 16:50:23.264361 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.60:41929
--webserver_interface=127.9.93.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.60.keytab
--principal=oryx/127.9.93.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:39869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:33733,127.9.93.61:38393,127.9.93.60:41929
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:50:23.273218 16327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.247350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52414 (local address 127.9.93.61:38393)
0928 16:50:23.247563 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:23.247571 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:23.252148 (+ 4577us) server_negotiation.cc:407] Connection header received
0928 16:50:23.252596 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:23.252601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:23.252707 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:23.252797 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:23.254102 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.255012 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.256542 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.257127 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.257260 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:23.261590 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:23.261600 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:23.261604 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:23.261647 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:23.269702 (+ 8055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:23.270813 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:23.270818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:23.270820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:23.270993 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:23.271474 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:23.271478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:23.271480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:23.271824 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:23.271991 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:23.272513 (+ 522us) server_negotiation.cc:299] Negotiation successful
0928 16:50:23.272795 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 16:50:23.273262 16352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.247288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:38393 (local address 127.0.0.1:52414)
0928 16:50:23.251963 (+ 4675us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:23.252018 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:23.252347 (+ 329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:23.252908 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:23.252916 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:23.253671 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:23.253922 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:23.253933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.255156 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.255162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:23.256380 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:23.256396 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.257316 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.257322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:23.257570 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:23.258344 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:23.258365 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:23.261338 (+ 2973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:23.269928 (+ 8590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:23.269947 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:23.269954 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:23.270642 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:23.271135 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:23.271141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:23.271143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:23.271337 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:23.271979 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:23.271989 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:23.272192 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 16:50:23.272366 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 16:50:23.272794 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4360,"thread_start_us":4164,"threads_started":1}
I20260928 16:50:23.276042 15789 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.93.61:38393: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "31a246c9c6804fdb8430171f57199a93"
member_type: NON_VOTER
last_known_addr {
host: "127.9.93.61"
port: 38393
}
attrs {
promote: true
}
}
dest_uuid: "2577aff545514d4389450abad8337d7b"
cas_config_opid_index: -1
I20260928 16:50:23.277561 15789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: NON_VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: true } }
I20260928 16:50:23.293869 16299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:23.293617 16356 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 31a246c9c6804fdb8430171f57199a93 (127.9.93.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: NON_VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: true } } }
I20260928 16:50:23.298424 16299 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 1 FOLLOWER]: Refusing update from remote peer 2577aff545514d4389450abad8337d7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 16:50:23.299103 16356 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.93.61:38393
I20260928 16:50:23.300354 16357 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: NON_VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:50:23.301352 16256 master.cc:537] Master@127.9.93.61:38393 shutting down...
W20260928 16:50:23.304764 15775 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260928 16:50:23.313442 16355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2577aff545514d4389450abad8337d7b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: NON_VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: true } } }
I20260928 16:50:23.313917 16355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [sys.catalog]: This master's current role is: LEADER
W20260928 16:50:23.315176 15775 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b -> Peer 31a246c9c6804fdb8430171f57199a93 (127.9.93.61:38393): Couldn't send request to peer 31a246c9c6804fdb8430171f57199a93. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 16:50:23.318058 16256 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:50:23.318589 16256 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:23.318810 16256 tablet_replica.cc:330] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93: stopping tablet replica
I20260928 16:50:23.326905 16256 master.cc:552] Master@127.9.93.61:38393 shutdown complete.
I20260928 16:50:23.327080 16256 master_runner.cc:305] Clearing existing system tablet
I20260928 16:50:23.327719 16256 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:23.327837 16256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:23.336223 16256 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.004s sys 0.004s
I20260928 16:50:23.338233 16362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:23.338677 16256 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:23.339190 16256 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 16:50:23.339381 16256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/wal
uuid: "31a246c9c6804fdb8430171f57199a93"
format_stamp: "Formatted at 2026-09-28 16:50:23 on dist-test-slave-wzcb"
server_key: "b25b70975d4abfef847f675aa3faef28"
server_key_iv: "227dc9c8d924d57f1b5b7fd84925e36e"
server_key_version: "encryptionkey@0"
I20260928 16:50:23.346269 16256 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:50:23.355262 16256 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 16:50:23.355677 16256 log.cc:1196] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 16:50:23.356338 16256 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93: Deleting consensus metadata
I20260928 16:50:23.356709 16256 master_runner.cc:315] Copying system tablet from 127.9.93.62:33733
I20260928 16:50:23.360010 16256 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93: tablet copy: Beginning tablet copy session from remote peer at address 127.9.93.62:33733
I20260928 16:50:23.377893 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.361314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32824 (local address 127.9.93.62:33733)
0928 16:50:23.361520 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:23.361526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:23.362047 (+ 521us) server_negotiation.cc:407] Connection header received
0928 16:50:23.362262 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:23.362269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:23.362402 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:23.362518 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:23.363749 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.364882 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.366857 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.367648 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.367761 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:23.370589 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:23.370596 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:23.370601 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:23.370645 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:23.374269 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:23.375249 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:23.375256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:23.375259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:23.375490 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:23.375989 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:23.375995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:23.375999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:23.376452 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:23.376578 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:23.377232 (+ 654us) server_negotiation.cc:299] Negotiation successful
0928 16:50:23.377502 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 16:50:23.379483 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.361209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.0.0.1:32824)
0928 16:50:23.361902 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:23.361923 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:23.362120 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:23.362568 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:23.362573 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:23.363183 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:23.363547 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:23.363563 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.365054 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.365060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:23.366646 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:23.366665 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.368109 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.368116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:23.368328 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:23.369062 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:23.369102 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:23.370298 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:23.374496 (+ 4198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:23.374503 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:23.374507 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:23.375083 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:23.375666 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:23.375671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:23.375676 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:23.375863 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:23.376678 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:23.376683 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:23.376883 (+ 200us) client_negotiation.cc:769] Sending connection context
0928 16:50:23.378520 (+ 1637us) client_negotiation.cc:240] Negotiation successful
0928 16:50:23.378964 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":416,"threads_started":1}
I20260928 16:50:23.380841 15810 tablet_copy_service.cc:140] P 2577aff545514d4389450abad8337d7b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 31a246c9c6804fdb8430171f57199a93 ({username='oryx', principal='oryx/127.9.93.61@KRBTEST.COM'} at 127.0.0.1:32824)
I20260928 16:50:23.381381 15810 tablet_copy_service.cc:161] P 2577aff545514d4389450abad8337d7b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 31a246c9c6804fdb8430171f57199a93 at {username='oryx', principal='oryx/127.9.93.61@KRBTEST.COM'} at 127.0.0.1:32824: session id = 31a246c9c6804fdb8430171f57199a93-00000000000000000000000000000000
I20260928 16:50:23.385385 15810 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 16:50:23.388341 16256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 16:50:23.394419 16256 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93: tablet copy: Starting download of 0 data blocks...
I20260928 16:50:23.394884 16256 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93: tablet copy: Starting download of 1 WAL segments...
I20260928 16:50:23.398761 16256 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 16:50:23.402837 15810 tablet_copy_service.cc:342] P 2577aff545514d4389450abad8337d7b: Request end of tablet copy session 31a246c9c6804fdb8430171f57199a93-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.93.61@KRBTEST.COM'} at 127.0.0.1:32824
I20260928 16:50:23.403133 15810 tablet_copy_service.cc:433] P 2577aff545514d4389450abad8337d7b: ending tablet copy session 31a246c9c6804fdb8430171f57199a93-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 31a246c9c6804fdb8430171f57199a93
I20260928 16:50:23.419139 16256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:23.422701 16369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:23.424022 16370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:23.441448 16372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:23.442056 16256 server_base.cc:1034] running on GCE node
I20260928 16:50:23.442461 16256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:23.443218 16256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:23.444496 16256 hybrid_clock.cc:648] HybridClock initialized: now 1790614223444430 us; error 33 us; skew 500 ppm
I20260928 16:50:23.444769 16256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:23 dist-test-slave-wzcb krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614223, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:23.447266 16256 init.cc:377] Logged in from keytab as oryx/127.9.93.61@KRBTEST.COM (short username oryx)
I20260928 16:50:23.451139 16256 webserver.cc:460] Webserver started at http://127.9.93.61:42713/ using document root <none> and password file <none>
I20260928 16:50:23.451609 16256 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:23.451713 16256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:23.455389 16256 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260928 16:50:23.457836 16379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:23.458343 16256 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:50:23.458523 16256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/wal
uuid: "31a246c9c6804fdb8430171f57199a93"
format_stamp: "Formatted at 2026-09-28 16:50:23 on dist-test-slave-wzcb"
server_key: "b25b70975d4abfef847f675aa3faef28"
server_key_iv: "227dc9c8d924d57f1b5b7fd84925e36e"
server_key_version: "encryptionkey@0"
I20260928 16:50:23.458738 16256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:23.470937 16256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:23.475585 16256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:23.502717 16431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.61:38393 every 8 connection(s)
I20260928 16:50:23.502755 16256 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.61:38393
I20260928 16:50:23.504189 16256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-1/data/info.pb
I20260928 16:50:23.510314 16432 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:50:23.514703 16432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93: Bootstrap starting.
I20260928 16:50:23.538147 16432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:50:23.538889 16432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93: Bootstrap complete.
I20260928 16:50:23.540227 16432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: NON_VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: true } }
I20260928 16:50:23.540572 16432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31a246c9c6804fdb8430171f57199a93, State: Initialized, Role: LEARNER
I20260928 16:50:23.540870 16432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: NON_VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: true } }
I20260928 16:50:23.542276 16432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:50:23.542013 16435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: NON_VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: true } } }
I20260928 16:50:23.542752 16435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [sys.catalog]: This master's current role is: LEARNER
I20260928 16:50:23.547366 16250 heartbeater.cc:503] Master 127.9.93.62:33733 was elected leader, sending a full tablet report...
I20260928 16:50:23.554900 16446 catalog_manager.cc:1259] Loaded cluster ID: 5bcd09d1c1b14d24a6129be96c99a994
I20260928 16:50:23.555042 16446 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93: loading cluster ID for follower catalog manager: success
I20260928 16:50:23.559186 16446 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93: acquiring CA information for follower catalog manager: success
I20260928 16:50:23.562294 16446 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 16:50:23.756443 16353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:23.756837 16353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:23.757006 16353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:23.765628 16353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:50:23.765802 16353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:23.765887 16353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:23.765969 16353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:50:23.766031 16353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:50:23.777369 16353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:33733,127.9.93.61:38393,127.9.93.60:41929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.60.keytab
--principal=oryx/127.9.93.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.60:41929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.93.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:23.779019 16353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:23.780673 16353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:23.794368 16452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:23.794349 16451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:23.794301 16454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:23.795406 16353 server_base.cc:1034] running on GCE node
I20260928 16:50:23.796401 16353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:23.798264 16353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:23.799475 16353 hybrid_clock.cc:648] HybridClock initialized: now 1790614223799403 us; error 69 us; skew 500 ppm
I20260928 16:50:23.799902 16353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:23 dist-test-slave-wzcb krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614223, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:23.807703 16353 init.cc:377] Logged in from keytab as oryx/127.9.93.60@KRBTEST.COM (short username oryx)
I20260928 16:50:23.809664 16353 webserver.cc:460] Webserver started at http://127.9.93.60:43969/ using document root <none> and password file <none>
I20260928 16:50:23.810555 16353 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:23.810717 16353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:23.811075 16353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:23.815028 16353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/data/instance:
uuid: "6cebda140f974905b3e3480b980f7a2d"
format_stamp: "Formatted at 2026-09-28 16:50:23 on dist-test-slave-wzcb"
server_key: "6eb453841b865e944281152966da0ea6"
server_key_iv: "9559ef73c2bb074d64ef9f396a37795d"
server_key_version: "encryptionkey@0"
I20260928 16:50:23.816226 16353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/wal/instance:
uuid: "6cebda140f974905b3e3480b980f7a2d"
format_stamp: "Formatted at 2026-09-28 16:50:23 on dist-test-slave-wzcb"
server_key: "6eb453841b865e944281152966da0ea6"
server_key_iv: "9559ef73c2bb074d64ef9f396a37795d"
server_key_version: "encryptionkey@0"
I20260928 16:50:23.822975 16353 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 16:50:23.827761 16461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:23.829627 16353 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:50:23.829879 16353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/wal
uuid: "6cebda140f974905b3e3480b980f7a2d"
format_stamp: "Formatted at 2026-09-28 16:50:23 on dist-test-slave-wzcb"
server_key: "6eb453841b865e944281152966da0ea6"
server_key_iv: "9559ef73c2bb074d64ef9f396a37795d"
server_key_version: "encryptionkey@0"
I20260928 16:50:23.830130 16353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:23.850643 16353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:23.855682 16353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:23.855986 16353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:23.876513 16353 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.60:41929
I20260928 16:50:23.876574 16512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.60:41929 every 8 connection(s)
I20260928 16:50:23.878193 16353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/data/info.pb
I20260928 16:50:23.882807 16513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:23.887734 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 16353
I20260928 16:50:23.892196 16513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:50:23 dist-test-slave-wzcb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.93.60@KRBTEST.COM
Sep 28 16:50:23 dist-test-slave-wzcb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614223, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.60@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM
I20260928 16:50:23.918133 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.896549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32826 (local address 127.9.93.62:33733)
0928 16:50:23.897151 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:23.897157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:23.897826 (+ 669us) server_negotiation.cc:407] Connection header received
0928 16:50:23.898060 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:23.898064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:23.898137 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:23.898240 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:23.899858 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.900879 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.902986 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.904030 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.905202 (+ 1172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:23.909741 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:23.909750 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:23.909755 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:23.909792 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:23.914261 (+ 4469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:23.915254 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:23.915260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:23.915263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:23.915490 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:23.915902 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:23.915907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:23.915911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:23.916208 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:23.916353 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:23.917516 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0928 16:50:23.917807 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":301,"threads_started":1}
I20260928 16:50:23.919502 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.896483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.0.0.1:32826)
0928 16:50:23.897627 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:23.897714 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:23.897938 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:23.898346 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:23.898354 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:23.899384 (+ 1030us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:23.899691 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:23.899702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.901061 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.901066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:23.902769 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:23.902787 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.904376 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.904384 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:23.904657 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:23.905527 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:23.905566 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:23.909309 (+ 3743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:23.914476 (+ 5167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:23.914493 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:23.914505 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:23.915109 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:23.915627 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:23.915632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:23.915635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:23.915791 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:23.916356 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:23.916365 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:23.916628 (+ 263us) client_negotiation.cc:769] Sending connection context
0928 16:50:23.916952 (+ 324us) client_negotiation.cc:240] Negotiation successful
0928 16:50:23.917701 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"mutex_wait_us":144,"thread_start_us":623,"threads_started":1}
I20260928 16:50:23.919574 16516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.890500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51866 (local address 127.9.93.60:41929)
0928 16:50:23.891501 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:23.891516 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:23.891568 (+ 52us) server_negotiation.cc:407] Connection header received
0928 16:50:23.893526 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:23.893556 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:23.894129 (+ 573us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:23.894601 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:23.896389 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.898679 (+ 2290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.900517 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.901533 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.901736 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:23.908743 (+ 7007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:23.908758 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:23.908782 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:23.908841 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:23.913407 (+ 4566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:23.914591 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:23.914606 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:23.914609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:23.914881 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:23.915475 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:23.915480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:23.915482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:23.916398 (+ 916us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:23.916736 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:23.917107 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 16:50:23.918474 (+ 1367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":448,"threads_started":1}
I20260928 16:50:23.921702 16513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:50:23.922425 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.902310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52420 (local address 127.9.93.61:38393)
0928 16:50:23.914567 (+ 12257us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:23.914573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:23.914606 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:50:23.914689 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:23.914695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:23.914809 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:23.914964 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:50:23.915703 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.917252 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.919940 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.921243 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.921402 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:23.921687 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:23.921916 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 16:50:23.922071 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12086,"thread_start_us":11928,"threads_started":1}
I20260928 16:50:23.922523 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.901787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:38393 (local address 127.0.0.1:52420)
0928 16:50:23.913553 (+ 11766us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:23.913577 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:23.913737 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:23.915070 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:23.915077 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:23.915110 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:50:23.915498 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:23.915510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.917442 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.917448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:23.919671 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:23.919690 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.921449 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.921455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:23.921659 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:23.921735 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 16:50:23.921833 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 16:50:23.922095 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11584,"thread_start_us":12051,"threads_started":1}
I20260928 16:50:23.923791 16407 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260928 16:50:23.924466 16435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2577aff545514d4389450abad8337d7b. Latest consensus state: current_term: 1 leader_uuid: "2577aff545514d4389450abad8337d7b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: NON_VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: true } } }
I20260928 16:50:23.924759 16435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [sys.catalog]: This master's current role is: LEARNER
Sep 28 16:50:23 dist-test-slave-wzcb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614223, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.60@KRBTEST.COM for oryx/127.9.93.61@KRBTEST.COM
I20260928 16:50:23.936890 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.923256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52428 (local address 127.9.93.61:38393)
0928 16:50:23.923963 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:23.923968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:23.923982 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:50:23.924023 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:23.924026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:23.924129 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:23.924225 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:23.925147 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.925900 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.927107 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.927600 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.927689 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:23.930979 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:23.930985 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:23.930989 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:23.931029 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:23.933955 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:23.934961 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:23.934965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:23.934968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:23.935126 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:23.935520 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:23.935523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:23.935525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:23.935906 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:23.936026 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:23.936379 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 16:50:23.936622 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572}
I20260928 16:50:23.936893 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.923199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:38393 (local address 127.0.0.1:52428)
0928 16:50:23.923561 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:23.923583 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:23.923691 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:23.924276 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:23.924281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:23.924774 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:23.924985 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:23.924998 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.926007 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.926011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:23.926990 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:23.927002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.927734 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.927739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:23.927913 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:23.928509 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:23.928529 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:23.930758 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:23.934145 (+ 3387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:23.934151 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:23.934155 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:23.934834 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:23.935234 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:23.935237 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:23.935240 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:23.935442 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:23.936004 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:23.936008 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:23.936174 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 16:50:23.936293 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 16:50:23.936586 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226}
I20260928 16:50:23.938196 16513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:50:23 dist-test-slave-wzcb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614223, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.60@KRBTEST.COM for oryx/127.9.93.60@KRBTEST.COM
I20260928 16:50:23.951951 16516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.938969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51876 (local address 127.9.93.60:41929)
0928 16:50:23.939130 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:23.939136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:23.939210 (+ 74us) server_negotiation.cc:407] Connection header received
0928 16:50:23.939398 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:23.939403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:23.939467 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:23.939546 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:23.940308 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.941053 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.942264 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.942703 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.942821 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:23.946387 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:23.946394 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:23.946400 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:23.946444 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:23.949192 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:23.949884 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:23.949889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:23.949893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:23.950030 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:23.950413 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:23.950418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:23.950422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:23.950773 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:23.950913 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:23.951257 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 16:50:23.951655 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":47}
I20260928 16:50:23.951951 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.938939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41929 (local address 127.0.0.1:51876)
0928 16:50:23.939130 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:23.939148 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:23.939269 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:23.939575 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:23.939579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:23.940013 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:23.940178 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:23.940187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.941179 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.941186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:23.942131 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:23.942144 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:23.942874 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:23.942880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:23.943102 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:23.943715 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:23.943732 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:23.946156 (+ 2424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:23.949355 (+ 3199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:23.949360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:23.949364 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:23.949766 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:23.950135 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:23.950141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:23.950145 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:23.950310 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:23.950887 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:23.950891 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:23.951035 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 16:50:23.951162 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 16:50:23.951494 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 16:50:23.957618 16513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d: Bootstrap starting.
I20260928 16:50:23.961675 16513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:23.962970 16513 log.cc:824] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d: Log is configured to *not* fsync() on all Append() calls
I20260928 16:50:23.966226 16513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d: No bootstrap required, opened a new log
I20260928 16:50:23.974967 16513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } }
I20260928 16:50:23.975497 16513 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:23.975643 16513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cebda140f974905b3e3480b980f7a2d, State: Initialized, Role: FOLLOWER
I20260928 16:50:23.976336 16513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } }
I20260928 16:50:23.977329 16523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } } }
I20260928 16:50:23.977680 16523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:50:23.978586 16513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [sys.catalog]: configured and running, proceeding with master startup.
W20260928 16:50:23.988965 16534 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:50:23.989164 16534 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:50:23.994174 16353 master_runner.cc:428] Detected that this master 6cebda140f974905b3e3480b980f7a2d is joining an existing cluster
I20260928 16:50:23.994319 16353 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.93.60:41929
I20260928 16:50:24.017989 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.998938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32828 (local address 127.9.93.62:33733)
0928 16:50:23.999160 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:23.999167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:24.000521 (+ 1354us) server_negotiation.cc:407] Connection header received
0928 16:50:24.000902 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:24.000909 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:24.001025 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:24.001100 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:24.003426 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.004371 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.006534 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.007248 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.007403 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:24.010544 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:24.010549 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:24.010553 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:24.010597 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:24.014086 (+ 3489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:24.015000 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:24.015007 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:24.015011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:24.015165 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:24.015736 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:24.015742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:24.015746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:24.016162 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:24.016323 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:24.017307 (+ 984us) server_negotiation.cc:299] Negotiation successful
0928 16:50:24.017603 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:50:24.018949 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.999551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.0.0.1:32828)
0928 16:50:24.000352 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:24.000375 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:24.000708 (+ 333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:24.001342 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:24.001345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:24.002474 (+ 1129us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:24.003194 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:24.003211 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.004571 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.004577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:24.006330 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:24.006349 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.007748 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.007755 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:24.007937 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:24.008805 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:24.008834 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:24.010275 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:24.014279 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:24.014284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:24.014288 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:24.014865 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:24.015298 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:24.015336 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:24.015340 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:24.015561 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:24.016401 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:24.016407 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:24.016706 (+ 299us) client_negotiation.cc:769] Sending connection context
0928 16:50:24.016864 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 16:50:24.018430 (+ 1566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":76,"thread_start_us":444,"threads_started":1}
I20260928 16:50:24.020890 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.999142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52436 (local address 127.9.93.61:38393)
0928 16:50:23.999396 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:23.999403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:24.001007 (+ 1604us) server_negotiation.cc:407] Connection header received
0928 16:50:24.001308 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:24.001314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:24.001406 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:24.001484 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:24.003844 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.005015 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.007749 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.008492 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.008607 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:24.011446 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:24.011453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:24.011458 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:24.011506 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:24.016754 (+ 5248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:24.017785 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:24.017792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:24.017796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:24.017978 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:24.018648 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:24.018654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:24.018658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:24.019218 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:24.019413 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:24.020131 (+ 718us) server_negotiation.cc:299] Negotiation successful
0928 16:50:24.020433 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 16:50:24.021837 16516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:24.001351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51886 (local address 127.9.93.60:41929)
0928 16:50:24.001537 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:24.001544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:24.004820 (+ 3276us) server_negotiation.cc:407] Connection header received
0928 16:50:24.005070 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:24.005074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:24.005144 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:24.005205 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:24.007103 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.008328 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.010256 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.011054 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.011188 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:24.013503 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:24.013509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:24.013514 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:24.013552 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:24.017721 (+ 4169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:24.018937 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:24.018943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:24.018946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:24.019295 (+ 349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:24.019750 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:24.019755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:24.019758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:24.020168 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:24.020319 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:24.020882 (+ 563us) server_negotiation.cc:299] Negotiation successful
0928 16:50:24.021416 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":37}
I20260928 16:50:24.022253 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:23.999733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:38393 (local address 127.0.0.1:52436)
0928 16:50:24.000886 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:24.000904 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:24.001140 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:24.001732 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:24.001736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:24.002484 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:24.003634 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:24.003649 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.005293 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.005299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:24.007521 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:24.007538 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.008971 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.008978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:24.009193 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:24.009891 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:24.009918 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:24.011147 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:24.016948 (+ 5801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:24.016953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:24.016956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:24.017622 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:24.018283 (+ 661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:24.018288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:24.018292 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:24.018498 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:24.019498 (+ 1000us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:24.019502 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:24.019793 (+ 291us) client_negotiation.cc:769] Sending connection context
0928 16:50:24.021211 (+ 1418us) client_negotiation.cc:240] Negotiation successful
0928 16:50:24.021595 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"mutex_wait_us":148,"spinlock_wait_cycles":42240,"thread_start_us":799,"threads_started":1}
I20260928 16:50:24.024200 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:24.001022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41929 (local address 127.0.0.1:51886)
0928 16:50:24.004712 (+ 3690us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:24.004733 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:24.004940 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:24.005383 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:24.005387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:24.006541 (+ 1154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:24.006944 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:24.006962 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.008502 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.008505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:24.010098 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:24.010116 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.011323 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.011331 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:24.011521 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:24.012097 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:24.012142 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:24.013277 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:24.017878 (+ 4601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:24.017883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:24.017887 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:24.018801 (+ 914us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:24.019466 (+ 665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:24.019471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:24.019474 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:24.019629 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:24.020407 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:24.020412 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:24.020614 (+ 202us) client_negotiation.cc:769] Sending connection context
0928 16:50:24.022278 (+ 1664us) client_negotiation.cc:240] Negotiation successful
0928 16:50:24.023740 (+ 1462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3479,"thread_start_us":2081,"threads_started":1}
I20260928 16:50:24.027575 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 15851
Sep 28 16:50:24 dist-test-slave-wzcb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614220, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.62@KRBTEST.COM for oryx/127.9.93.60@KRBTEST.COM
Sep 28 16:50:24 dist-test-slave-wzcb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.93.1@KRBTEST.COM
I20260928 16:50:24.044070 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:24.025831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41929 (local address 127.0.0.1:51900)
0928 16:50:24.026005 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:24.026028 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:24.026162 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:24.026705 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:24.026711 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:24.027347 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:24.027570 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:24.027584 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.029261 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.029267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:24.030990 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:24.031011 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.032310 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.032317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:24.032524 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:24.033291 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:24.033316 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:24.035823 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:24.040740 (+ 4917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:24.040748 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:24.040753 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:24.041412 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:24.042073 (+ 661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:24.042078 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:24.042080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:24.042246 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:24.042954 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:24.042960 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:24.043153 (+ 193us) client_negotiation.cc:769] Sending connection context
0928 16:50:24.043285 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 16:50:24.043675 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 16:50:24.044433 16516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:24.025901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51900 (local address 127.9.93.60:41929)
0928 16:50:24.026100 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:24.026106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:24.026128 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:50:24.026349 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:24.026354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:24.026495 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:24.027746 (+ 1251us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:24.027966 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.029073 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.031229 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.032112 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.032268 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:24.036103 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:24.036112 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:24.036118 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:24.036168 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:24.040490 (+ 4322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:24.041685 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:24.041692 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:24.041696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:24.041897 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:24.042418 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:24.042423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:24.042426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:24.042811 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:24.042995 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:24.043767 (+ 772us) server_negotiation.cc:299] Negotiation successful
0928 16:50:24.044083 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 16:50:24.046437 15789 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.93.60:41929: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6cebda140f974905b3e3480b980f7a2d"
member_type: NON_VOTER
last_known_addr {
host: "127.9.93.60"
port: 41929
}
attrs {
promote: true
}
}
dest_uuid: "2577aff545514d4389450abad8337d7b"
cas_config_opid_index: 5
I20260928 16:50:24.047499 15789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: NON_VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: true } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: NON_VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: true } }
I20260928 16:50:24.049365 16519 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6cebda140f974905b3e3480b980f7a2d (127.9.93.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: NON_VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: true } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: NON_VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: true } } }
I20260928 16:50:24.050379 16407 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 1 LEARNER]: Refusing update from remote peer 2577aff545514d4389450abad8337d7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 16:50:24.053018 16488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:24.053141 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:24.028892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52582 (local address 127.9.93.1:40577)
0928 16:50:24.029602 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:24.029610 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:24.029638 (+ 28us) server_negotiation.cc:407] Connection header received
0928 16:50:24.029700 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:24.029707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:24.029942 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:24.030159 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:24.031843 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.033411 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.035893 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.036576 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.036719 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:24.042272 (+ 5553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:24.042298 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:24.042323 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:24.042430 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:24.047677 (+ 5247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:24.048837 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:24.048850 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:24.048853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:24.049093 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:24.049749 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:24.049756 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:24.049760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:24.050404 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:24.050756 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:24.052298 (+ 1542us) server_negotiation.cc:299] Negotiation successful
0928 16:50:24.052669 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":360,"threads_started":1}
I20260928 16:50:24.053610 16519 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.93.60:41929
I20260928 16:50:24.054747 15975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:24.068255 16519 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: NON_VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 16:50:24.069115 16519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2577aff545514d4389450abad8337d7b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: NON_VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: true } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: NON_VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: true } } }
I20260928 16:50:24.069484 16519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:24.072553 16407 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6cebda140f974905b3e3480b980f7a2d (127.9.93.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: NON_VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: true } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: NON_VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: true } } }
I20260928 16:50:24.075982 16353 master.cc:537] Master@127.9.93.60:41929 shutting down...
I20260928 16:50:24.077095 16488 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 1 FOLLOWER]: Refusing update from remote peer 2577aff545514d4389450abad8337d7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 16:50:24.078306 16545 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: NON_VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:50:24.085886 16523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:50:24.086221 16523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } }
I20260928 16:50:24.090139 16523 leader_election.cc:288] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2577aff545514d4389450abad8337d7b (127.9.93.62:33733), 31a246c9c6804fdb8430171f57199a93 (127.9.93.61:38393)
I20260928 16:50:24.091398 15799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cebda140f974905b3e3480b980f7a2d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2577aff545514d4389450abad8337d7b" is_pre_election: true
I20260928 16:50:24.093675 16406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cebda140f974905b3e3480b980f7a2d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a246c9c6804fdb8430171f57199a93" is_pre_election: true
I20260928 16:50:24.094789 16464 leader_election.cc:302] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cebda140f974905b3e3480b980f7a2d; no voters: 2577aff545514d4389450abad8337d7b, 31a246c9c6804fdb8430171f57199a93
I20260928 16:50:24.095739 16523 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:50:24.095744 16435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2577aff545514d4389450abad8337d7b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: NON_VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: true } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: NON_VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: true } } }
I20260928 16:50:24.096076 16435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [sys.catalog]: This master's current role is: LEARNER
I20260928 16:50:24.097674 16353 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:50:24.098256 16353 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:24.098519 16353 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d: stopping tablet replica
I20260928 16:50:24.100481 16544 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b: attempting to promote NON_VOTER 31a246c9c6804fdb8430171f57199a93 to VOTER
I20260928 16:50:24.101905 16544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: NON_VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: true } }
I20260928 16:50:24.105669 16407 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 1 LEARNER]: Refusing update from remote peer 2577aff545514d4389450abad8337d7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 16:50:24.106346 16545 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 16:50:24.106608 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:24.105995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41929 (local address 127.0.0.1:51912)
0928 16:50:24.106245 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:24.106385 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260928 16:50:24.107234 15775 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b -> Peer 6cebda140f974905b3e3480b980f7a2d (127.9.93.60:41929): Couldn't send request to peer 6cebda140f974905b3e3480b980f7a2d. Status: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 16:50:24.110378 16519 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 31a246c9c6804fdb8430171f57199a93 (127.9.93.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: NON_VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: true } } }
I20260928 16:50:24.113540 16353 master.cc:552] Master@127.9.93.60:41929 shutdown complete.
I20260928 16:50:24.113739 16353 master_runner.cc:305] Clearing existing system tablet
I20260928 16:50:24.114257 16353 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:24.114415 16353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:24.115602 16545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2577aff545514d4389450abad8337d7b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: NON_VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: true } } }
I20260928 16:50:24.115908 16545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:24.116362 16545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2577aff545514d4389450abad8337d7b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: NON_VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: true } } }
I20260928 16:50:24.116634 16545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:24.117400 16407 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 31a246c9c6804fdb8430171f57199a93 (127.9.93.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: NON_VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: true } } }
I20260928 16:50:24.122336 16435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2577aff545514d4389450abad8337d7b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: NON_VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: true } } }
I20260928 16:50:24.122671 16435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:50:24.124200 16353 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.006s sys 0.001s
I20260928 16:50:24.126300 16551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:24.126768 16353 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:24.127153 16353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 16:50:24.127333 16353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/wal
uuid: "6cebda140f974905b3e3480b980f7a2d"
format_stamp: "Formatted at 2026-09-28 16:50:23 on dist-test-slave-wzcb"
server_key: "6eb453841b865e944281152966da0ea6"
server_key_iv: "9559ef73c2bb074d64ef9f396a37795d"
server_key_version: "encryptionkey@0"
I20260928 16:50:24.133333 16353 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:50:24.141923 16353 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 16:50:24.142234 16353 log.cc:1196] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 16:50:24.142865 16353 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d: Deleting consensus metadata
I20260928 16:50:24.143222 16353 master_runner.cc:315] Copying system tablet from 127.9.93.62:33733
I20260928 16:50:24.146685 16353 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d: tablet copy: Beginning tablet copy session from remote peer at address 127.9.93.62:33733
I20260928 16:50:24.174284 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:24.147814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32830 (local address 127.9.93.62:33733)
0928 16:50:24.148027 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:24.148033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:24.156156 (+ 8123us) server_negotiation.cc:407] Connection header received
0928 16:50:24.156377 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:24.156384 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:24.156514 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:24.156642 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:24.158090 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.159441 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.162240 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.163189 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.163444 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:24.166449 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:24.166466 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:24.166471 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:24.166524 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:24.170511 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:24.171666 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:24.171674 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:24.171678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:24.171889 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:24.172447 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:24.172461 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:24.172464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:24.172877 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:24.173037 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:24.173595 (+ 558us) server_negotiation.cc:299] Negotiation successful
0928 16:50:24.173879 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 16:50:24.174398 16557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:24.147796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.0.0.1:32830)
0928 16:50:24.155943 (+ 8147us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:24.155979 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:24.156221 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:24.156706 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:24.156712 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:24.157461 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:24.157827 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:24.157846 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.159743 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.159749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:24.161895 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:24.161936 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.163514 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.163522 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:24.163781 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:24.164675 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:24.164708 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:24.166103 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:24.170757 (+ 4654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:24.170765 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:24.170770 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:24.171482 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:24.172071 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:24.172077 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:24.172080 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:24.172318 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:24.173044 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:24.173054 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:24.173289 (+ 235us) client_negotiation.cc:769] Sending connection context
0928 16:50:24.173458 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 16:50:24.173945 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7954,"thread_start_us":7789,"threads_started":1}
I20260928 16:50:24.175364 15810 tablet_copy_service.cc:140] P 2577aff545514d4389450abad8337d7b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6cebda140f974905b3e3480b980f7a2d ({username='oryx', principal='oryx/127.9.93.60@KRBTEST.COM'} at 127.0.0.1:32830)
I20260928 16:50:24.175580 15810 tablet_copy_service.cc:161] P 2577aff545514d4389450abad8337d7b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6cebda140f974905b3e3480b980f7a2d at {username='oryx', principal='oryx/127.9.93.60@KRBTEST.COM'} at 127.0.0.1:32830: session id = 6cebda140f974905b3e3480b980f7a2d-00000000000000000000000000000000
I20260928 16:50:24.180084 15810 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 16:50:24.182750 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 15851
I20260928 16:50:24.183514 16353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 16:50:24.192541 16353 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d: tablet copy: Starting download of 0 data blocks...
I20260928 16:50:24.193003 16353 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d: tablet copy: Starting download of 1 WAL segments...
I20260928 16:50:24.196444 16353 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 16:50:24.201452 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:40577
--local_ip_for_outbound_sockets=127.9.93.1
--tserver_master_addrs=127.9.93.62:33733
--webserver_port=41979
--webserver_interface=127.9.93.1
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.1.keytab
--principal=oryx/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:39869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.93.62:33733,127.9.93.61:38393
--tserver_master_addrs=127.9.93.62:33733,127.9.93.61:38393,127.9.93.60:41929 with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:50:24.202219 15810 tablet_copy_service.cc:342] P 2577aff545514d4389450abad8337d7b: Request end of tablet copy session 6cebda140f974905b3e3480b980f7a2d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.93.60@KRBTEST.COM'} at 127.0.0.1:32830
I20260928 16:50:24.202401 15810 tablet_copy_service.cc:433] P 2577aff545514d4389450abad8337d7b: ending tablet copy session 6cebda140f974905b3e3480b980f7a2d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6cebda140f974905b3e3480b980f7a2d
I20260928 16:50:24.216861 16353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:24.219949 16559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:24.220605 16560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:24.225812 16353 server_base.cc:1034] running on GCE node
W20260928 16:50:24.226493 16562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:24.226943 16353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:24.227900 16353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:24.229106 16353 hybrid_clock.cc:648] HybridClock initialized: now 1790614224229060 us; error 33 us; skew 500 ppm
I20260928 16:50:24.229367 16353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:24 dist-test-slave-wzcb krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614224, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:24.231366 16353 init.cc:377] Logged in from keytab as oryx/127.9.93.60@KRBTEST.COM (short username oryx)
I20260928 16:50:24.236367 16353 webserver.cc:460] Webserver started at http://127.9.93.60:36909/ using document root <none> and password file <none>
I20260928 16:50:24.236698 16353 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:24.236811 16353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:24.239589 16353 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:50:24.241412 16569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:24.241914 16353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260928 16:50:24.242050 16353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/wal
uuid: "6cebda140f974905b3e3480b980f7a2d"
format_stamp: "Formatted at 2026-09-28 16:50:23 on dist-test-slave-wzcb"
server_key: "6eb453841b865e944281152966da0ea6"
server_key_iv: "9559ef73c2bb074d64ef9f396a37795d"
server_key_version: "encryptionkey@0"
I20260928 16:50:24.242244 16353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:24.288002 16353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:24.291874 16353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:24.321463 16353 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.60:41929
I20260928 16:50:24.321506 16620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.60:41929 every 8 connection(s)
I20260928 16:50:24.322513 16353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/master-2/data/info.pb
I20260928 16:50:24.329216 16621 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:50:24.332360 16621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d: Bootstrap starting.
I20260928 16:50:24.350214 16621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:50:24.350975 16621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d: Bootstrap complete.
I20260928 16:50:24.352298 16621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: NON_VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: true } }
I20260928 16:50:24.352623 16621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6cebda140f974905b3e3480b980f7a2d, State: Initialized, Role: LEARNER
I20260928 16:50:24.352943 16621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: NON_VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: true } }
I20260928 16:50:24.353873 16624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: NON_VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: true } } }
I20260928 16:50:24.354259 16621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:50:24.354471 16624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [sys.catalog]: This master's current role is: LEARNER
I20260928 16:50:24.365872 16635 catalog_manager.cc:1259] Loaded cluster ID: 5bcd09d1c1b14d24a6129be96c99a994
I20260928 16:50:24.366145 16635 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d: loading cluster ID for follower catalog manager: success
I20260928 16:50:24.370450 16635 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d: acquiring CA information for follower catalog manager: success
I20260928 16:50:24.373729 16635 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 16:50:24.642777 16558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:24.643178 16558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:24.643352 16558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:24.654335 16558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:24.654517 16558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:24.654728 16558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:50:24.667640 16639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:24.649493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41929 (local address 127.0.0.1:51914)
0928 16:50:24.661473 (+ 11980us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:24.661497 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:24.661696 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:24.662153 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:24.662159 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:24.662189 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:50:24.662450 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:24.662460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.663877 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.663881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:24.665217 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:24.665232 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.666715 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.666722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:24.666914 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:24.666952 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 16:50:24.667053 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 16:50:24.667246 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11852,"thread_start_us":11730,"threads_started":1}
I20260928 16:50:24.667656 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:24.649713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51914 (local address 127.9.93.60:41929)
0928 16:50:24.660917 (+ 11204us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:24.660933 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:24.661637 (+ 704us) server_negotiation.cc:407] Connection header received
0928 16:50:24.661856 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:24.661861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:24.661958 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:24.662096 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:50:24.662622 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.663713 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.665532 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.666474 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.666622 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:24.666917 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:24.667094 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 16:50:24.667280 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10973,"thread_start_us":11211,"threads_started":1}
I20260928 16:50:24.668797 16596 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 16:50:24.669492 16624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2577aff545514d4389450abad8337d7b. Latest consensus state: current_term: 1 leader_uuid: "2577aff545514d4389450abad8337d7b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: NON_VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: true } } }
I20260928 16:50:24.669821 16624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [sys.catalog]: This master's current role is: LEARNER
I20260928 16:50:24.672483 16558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.1.keytab
--principal=oryx/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:40577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=41979
--tserver_master_addrs=127.9.93.62:33733,127.9.93.61:38393,127.9.93.60:41929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:24.674031 16558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:24.675748 16558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:24.691907 16643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:24.691838 16645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:24.691843 16642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:24.692909 16558 server_base.cc:1034] running on GCE node
I20260928 16:50:24.693706 16558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:24.694976 16558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:24.696255 16558 hybrid_clock.cc:648] HybridClock initialized: now 1790614224696176 us; error 84 us; skew 500 ppm
I20260928 16:50:24.696697 16558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:24 dist-test-slave-wzcb krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614224, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:24.700727 16558 init.cc:377] Logged in from keytab as oryx/127.9.93.1@KRBTEST.COM (short username oryx)
I20260928 16:50:24.702906 16558 webserver.cc:460] Webserver started at http://127.9.93.1:41979/ using document root <none> and password file <none>
I20260928 16:50:24.703933 16558 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:24.704128 16558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:24.711115 16558 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 16:50:24.714896 16652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:24.716931 16558 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:50:24.717186 16558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "8c5622e16bb5462793db42b715d8d31b"
format_stamp: "Formatted at 2026-09-28 16:50:21 on dist-test-slave-wzcb"
server_key: "8b9bf11015244de2cfef9ae45ac71c89"
server_key_iv: "a87ddf20c84ce13dde202bc96dc44168"
server_key_version: "encryptionkey@0"
I20260928 16:50:24.718819 16558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:24.741842 16558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:24.747395 16558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:24.747710 16558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:24.748898 16558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:50:24.750633 16558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:24.750741 16558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:24.750869 16558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:24.750941 16558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:24.784827 16558 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:40577
I20260928 16:50:24.784900 16764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:40577 every 8 connection(s)
I20260928 16:50:24.786612 16558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:50:24.788547 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 16558
I20260928 16:50:24.788978 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 15986
Sep 28 16:50:24 dist-test-slave-wzcb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.93.2@KRBTEST.COM
Sep 28 16:50:24 dist-test-slave-wzcb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614224, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.60@KRBTEST.COM
Sep 28 16:50:24 dist-test-slave-wzcb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614224, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM
I20260928 16:50:24.829501 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:24.791214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:32891 (local address 127.9.93.60:41929)
0928 16:50:24.791533 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:24.791540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:24.802583 (+ 11043us) server_negotiation.cc:407] Connection header received
0928 16:50:24.806261 (+ 3678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:24.806271 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:24.806405 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:24.806488 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:24.809823 (+ 3335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.811018 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.814046 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.814786 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.814907 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:24.820858 (+ 5951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:24.820866 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:24.820871 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:24.820925 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:24.824607 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:24.825850 (+ 1243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:24.825858 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:24.825861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:24.826086 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:24.826878 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:24.826884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:24.826887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:24.827388 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:24.827553 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:24.828735 (+ 1182us) server_negotiation.cc:299] Negotiation successful
0928 16:50:24.829093 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260928 16:50:24.830657 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:24.792022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54740 (local address 127.9.93.2:43275)
0928 16:50:24.799837 (+ 7815us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:24.799851 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:24.799907 (+ 56us) server_negotiation.cc:407] Connection header received
0928 16:50:24.800021 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:24.800033 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:24.800453 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:24.803213 (+ 2760us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:24.804901 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.806948 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.809713 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.810744 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.818215 (+ 7471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:24.818633 (+ 418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:24.819278 (+ 645us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:24.819330 (+ 52us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:24.819395 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:24.824662 (+ 5267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:24.825881 (+ 1219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:24.825893 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:24.825897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:24.826128 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:24.827249 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:24.827256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:24.827259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:24.827853 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:24.828233 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:24.829805 (+ 1572us) server_negotiation.cc:299] Negotiation successful
0928 16:50:24.830188 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6839,"thread_start_us":6627,"threads_started":1}
I20260928 16:50:24.831354 16769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:24.790659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:35609 (local address 127.9.93.62:33733)
0928 16:50:24.801142 (+ 10483us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:24.801150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:24.808643 (+ 7493us) server_negotiation.cc:407] Connection header received
0928 16:50:24.810030 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:24.810037 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:24.810143 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:24.810228 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:24.811836 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.813108 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.815996 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.816746 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.816865 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:24.821900 (+ 5035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:24.821906 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:24.821909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:24.821947 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:24.826088 (+ 4141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:24.827128 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:24.827135 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:24.827155 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:24.827378 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:24.827910 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:24.827915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:24.827918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:24.828314 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:24.828459 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:24.830029 (+ 1570us) server_negotiation.cc:299] Negotiation successful
0928 16:50:24.830643 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10318,"thread_start_us":10180,"threads_started":1}
I20260928 16:50:24.832108 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:24.791472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.1:35609)
0928 16:50:24.808490 (+ 17018us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:24.808516 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:24.809852 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:24.810468 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:24.810474 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:24.811201 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:24.811601 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:24.811616 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.813360 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.813368 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:24.815762 (+ 2394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:24.815784 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.817250 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.817258 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:24.817492 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:24.818544 (+ 1052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:24.818574 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:24.821400 (+ 2826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:24.826253 (+ 4853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:24.826258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:24.826261 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:24.826953 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:24.827531 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:24.827536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:24.827540 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:24.827760 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:24.828523 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:24.828528 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:24.828701 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 16:50:24.828869 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 16:50:24.829551 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16840,"thread_start_us":10341,"threads_started":1}
I20260928 16:50:24.832145 16771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:24.791117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41929 (local address 127.9.93.1:32891)
0928 16:50:24.802355 (+ 11238us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:24.802435 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:24.805715 (+ 3280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:24.807337 (+ 1622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:24.807357 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:24.808433 (+ 1076us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:24.809561 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:24.809594 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.811257 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.811263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:24.813790 (+ 2527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:24.813811 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.815411 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.815421 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:24.815806 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:24.816820 (+ 1014us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:24.816854 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:24.820456 (+ 3602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:24.824847 (+ 4391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:24.824858 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:24.824880 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:24.825672 (+ 792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:24.826267 (+ 595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:24.826272 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:24.826275 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:24.826704 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:24.827670 (+ 966us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:24.827680 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:24.828233 (+ 553us) client_negotiation.cc:769] Sending connection context
0928 16:50:24.829739 (+ 1506us) client_negotiation.cc:240] Negotiation successful
0928 16:50:24.830115 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10843,"thread_start_us":18074,"threads_started":1}
I20260928 16:50:24.832425 16110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:24.834055 16766 heartbeater.cc:347] Connected to a master server at 127.9.93.62:33733
I20260928 16:50:24.834399 16765 heartbeater.cc:347] Connected to a master server at 127.9.93.60:41929
I20260928 16:50:24.834503 16766 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:24.834568 16765 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:24.835865 16765 heartbeater.cc:511] Master 127.9.93.60:41929 requested a full tablet report, sending...
I20260928 16:50:24.835893 16766 heartbeater.cc:511] Master 127.9.93.62:33733 requested a full tablet report, sending...
Sep 28 16:50:24 dist-test-slave-wzcb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614224, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.61@KRBTEST.COM
I20260928 16:50:24.838169 15789 ts_manager.cc:194] Re-registered known tserver with Master: 8c5622e16bb5462793db42b715d8d31b (127.9.93.1:40577)
I20260928 16:50:24.839330 16586 ts_manager.cc:194] Registered new tserver with Master: 8c5622e16bb5462793db42b715d8d31b (127.9.93.1:40577)
I20260928 16:50:24.839468 15789 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.93.1@KRBTEST.COM'} at 127.9.93.1:35609
I20260928 16:50:24.844887 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:24.816258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:38393 (local address 127.9.93.1:45125)
0928 16:50:24.820739 (+ 4481us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:24.822206 (+ 1467us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:24.822866 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:24.823671 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:24.823676 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:24.824358 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:24.824776 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:24.824792 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.826391 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.826397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:24.828191 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:24.828206 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.829579 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.829587 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:24.829978 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:24.831326 (+ 1348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:24.831353 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:24.837203 (+ 5850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:24.841808 (+ 4605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:24.841814 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:24.841819 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:24.842394 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:24.842911 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:24.842916 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:24.842919 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:24.843134 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:24.843940 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:24.843944 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:24.844083 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 16:50:24.844197 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 16:50:24.844525 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4347,"thread_start_us":4060,"threads_started":1}
I20260928 16:50:24.845068 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:24.810164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:45125 (local address 127.9.93.61:38393)
0928 16:50:24.818309 (+ 8145us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:24.818316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:24.822315 (+ 3999us) server_negotiation.cc:407] Connection header received
0928 16:50:24.823077 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:24.823083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:24.823206 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:24.823336 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:24.824998 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.826171 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.828394 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:24.829115 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:24.829228 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:24.837508 (+ 8280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:24.837517 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:24.837522 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:24.837581 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:24.841603 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:24.842573 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:24.842580 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:24.842591 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:24.842781 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:24.843258 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:24.843336 (+ 78us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:24.843340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:24.843789 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:24.843955 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:24.844339 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 16:50:24.844637 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7979,"thread_start_us":7832,"threads_started":1}
I20260928 16:50:24.846069 16767 heartbeater.cc:347] Connected to a master server at 127.9.93.61:38393
I20260928 16:50:24.846271 16767 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:24.846824 16767 heartbeater.cc:511] Master 127.9.93.61:38393 requested a full tablet report, sending...
I20260928 16:50:24.849081 16396 ts_manager.cc:194] Registered new tserver with Master: 8c5622e16bb5462793db42b715d8d31b (127.9.93.1:40577)
I20260928 16:50:24.927671 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 15986
I20260928 16:50:24.944167 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:43275
--local_ip_for_outbound_sockets=127.9.93.2
--tserver_master_addrs=127.9.93.62:33733
--webserver_port=43309
--webserver_interface=127.9.93.2
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.2.keytab
--principal=oryx/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:39869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.93.62:33733,127.9.93.61:38393
--tserver_master_addrs=127.9.93.62:33733,127.9.93.61:38393,127.9.93.60:41929 with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:50:25.137914 16519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2577aff545514d4389450abad8337d7b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: NON_VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: true } } }
I20260928 16:50:25.138306 16519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:25.140388 16778 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b: attempting to promote NON_VOTER 6cebda140f974905b3e3480b980f7a2d to VOTER
I20260928 16:50:25.141126 16778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: false } }
I20260928 16:50:25.143942 16407 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 1 FOLLOWER]: Refusing update from remote peer 2577aff545514d4389450abad8337d7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 16:50:25.144379 16596 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 1 LEARNER]: Refusing update from remote peer 2577aff545514d4389450abad8337d7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 16:50:25.144671 16779 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 16:50:25.145211 16519 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 16:50:25.148866 16519 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6cebda140f974905b3e3480b980f7a2d (127.9.93.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: false } } }
I20260928 16:50:25.149850 16407 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6cebda140f974905b3e3480b980f7a2d (127.9.93.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: false } } }
I20260928 16:50:25.152266 16596 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6cebda140f974905b3e3480b980f7a2d (127.9.93.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: false } } }
I20260928 16:50:25.154520 16779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2577aff545514d4389450abad8337d7b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: false } } }
I20260928 16:50:25.154918 16779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:25.155186 16779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2577aff545514d4389450abad8337d7b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: false } } }
I20260928 16:50:25.155506 16779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:25.154670 16519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2577aff545514d4389450abad8337d7b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: false } } }
I20260928 16:50:25.155969 16519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:25.159502 16624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2577aff545514d4389450abad8337d7b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: false } } }
I20260928 16:50:25.159818 16624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:50:25.161345 16784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2577aff545514d4389450abad8337d7b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: false } } }
I20260928 16:50:25.161900 16784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 16:50:25.338662 16777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:25.339022 16777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:25.339148 16777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:25.348031 16777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:25.348179 16777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:25.348325 16777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:50:25.359884 16777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.2.keytab
--principal=oryx/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:43275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=43309
--tserver_master_addrs=127.9.93.62:33733,127.9.93.61:38393,127.9.93.60:41929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:25.361564 16777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:25.363144 16777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:25.377954 16789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:25.378460 16790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:25.378499 16792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:25.378973 16777 server_base.cc:1034] running on GCE node
I20260928 16:50:25.379758 16777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:25.380944 16777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:25.382218 16777 hybrid_clock.cc:648] HybridClock initialized: now 1790614225382152 us; error 48 us; skew 500 ppm
I20260928 16:50:25.382629 16777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:25 dist-test-slave-wzcb krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614225, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:25.386296 16777 init.cc:377] Logged in from keytab as oryx/127.9.93.2@KRBTEST.COM (short username oryx)
I20260928 16:50:25.388172 16777 webserver.cc:460] Webserver started at http://127.9.93.2:43309/ using document root <none> and password file <none>
I20260928 16:50:25.389021 16777 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:25.389176 16777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:25.395882 16777 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:50:25.399452 16799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:25.401207 16777 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:50:25.401417 16777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "d29db76b68474ab6aa678633dd238fe1"
format_stamp: "Formatted at 2026-09-28 16:50:21 on dist-test-slave-wzcb"
server_key: "990362f1b065631a66346e900fedee7e"
server_key_iv: "5342072b0630725b4f35b03e661039ca"
server_key_version: "encryptionkey@0"
I20260928 16:50:25.402617 16777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:25.424829 16777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:25.429719 16777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:25.430040 16777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:25.431197 16777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:50:25.432847 16777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:25.432953 16777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:25.433072 16777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:25.433154 16777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:25.468068 16777 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:43275
I20260928 16:50:25.468135 16911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:43275 every 8 connection(s)
I20260928 16:50:25.469746 16777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:50:25.470837 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 16777
I20260928 16:50:25.471174 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 16121
Sep 28 16:50:25 dist-test-slave-wzcb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.93.3@KRBTEST.COM
Sep 28 16:50:25 dist-test-slave-wzcb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614225, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM
Sep 28 16:50:25 dist-test-slave-wzcb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614225, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.61@KRBTEST.COM
Sep 28 16:50:25 dist-test-slave-wzcb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614225, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.60@KRBTEST.COM
I20260928 16:50:25.519042 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:25.492947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:53333 (local address 127.9.93.61:38393)
0928 16:50:25.497400 (+ 4453us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:25.497409 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:25.497886 (+ 477us) server_negotiation.cc:407] Connection header received
0928 16:50:25.497973 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:25.497978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:25.498057 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:25.498160 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:25.499363 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:25.500529 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:25.502516 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:25.503215 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:25.504233 (+ 1018us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:25.509792 (+ 5559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:25.509800 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:25.509805 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:25.509876 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:25.513883 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:25.515163 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:25.515169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:25.515174 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:25.515398 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:25.516115 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:25.516120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:25.516124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:25.516617 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:25.516759 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:25.518378 (+ 1619us) server_negotiation.cc:299] Negotiation successful
0928 16:50:25.518684 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4301,"thread_start_us":4102,"threads_started":1}
I20260928 16:50:25.520511 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:25.473591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:40247 (local address 127.9.93.60:41929)
0928 16:50:25.486174 (+ 12583us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:25.486182 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:25.488789 (+ 2607us) server_negotiation.cc:407] Connection header received
0928 16:50:25.492406 (+ 3617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:25.492413 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:25.492526 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:25.492649 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:25.496850 (+ 4201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:25.499659 (+ 2809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:25.503412 (+ 3753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:25.504176 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:25.504313 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:25.511565 (+ 7252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:25.511574 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:25.511580 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:25.511633 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:25.515367 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:25.516439 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:25.516447 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:25.516450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:25.516640 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:25.517151 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:25.517156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:25.517160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:25.517559 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:25.517720 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:25.519820 (+ 2100us) server_negotiation.cc:299] Negotiation successful
0928 16:50:25.520119 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12401,"thread_start_us":12602,"threads_started":1}
I20260928 16:50:25.520551 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:25.474396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41929 (local address 127.9.93.2:40247)
0928 16:50:25.488561 (+ 14165us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:25.488625 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:25.491704 (+ 3079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:25.493232 (+ 1528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:25.493249 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:25.496253 (+ 3004us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:25.496592 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:25.496608 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:25.499860 (+ 3252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:25.499866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:25.502192 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:25.502209 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:25.504638 (+ 2429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:25.504645 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:25.504846 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:25.505470 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:25.505497 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:25.509479 (+ 3982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:25.515558 (+ 6079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:25.515565 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:25.515569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:25.516255 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:25.516793 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:25.516799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:25.516802 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:25.517001 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:25.517805 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:25.517811 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:25.517993 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 16:50:25.518153 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 16:50:25.518695 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13781,"mutex_wait_us":219,"spinlock_wait_cycles":39808,"thread_start_us":17641,"threads_started":1}
I20260928 16:50:25.520581 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:25.493767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:38393 (local address 127.9.93.2:53333)
0928 16:50:25.497643 (+ 3876us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:25.497663 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:25.497851 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:25.498214 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:25.498219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:25.498750 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:25.499128 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:25.499142 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:25.500716 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:25.500722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:25.502360 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:25.502371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:25.503701 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:25.503707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:25.503904 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:25.504542 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:25.504568 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:25.509525 (+ 4957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:25.514138 (+ 4613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:25.514150 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:25.514173 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:25.514985 (+ 812us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:25.515561 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:25.515567 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:25.515570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:25.515963 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:25.516849 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:25.516858 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:25.517614 (+ 756us) client_negotiation.cc:769] Sending connection context
0928 16:50:25.518012 (+ 398us) client_negotiation.cc:240] Negotiation successful
0928 16:50:25.519681 (+ 1669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3743,"thread_start_us":3640,"threads_started":1}
I20260928 16:50:25.522585 16914 heartbeater.cc:347] Connected to a master server at 127.9.93.61:38393
I20260928 16:50:25.522610 16912 heartbeater.cc:347] Connected to a master server at 127.9.93.60:41929
I20260928 16:50:25.522919 16914 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:25.522926 16912 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:25.524080 16912 heartbeater.cc:511] Master 127.9.93.60:41929 requested a full tablet report, sending...
I20260928 16:50:25.524108 16914 heartbeater.cc:511] Master 127.9.93.61:38393 requested a full tablet report, sending...
I20260928 16:50:25.525015 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:25.472621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37798 (local address 127.9.93.3:42893)
0928 16:50:25.483346 (+ 10725us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:25.483359 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:25.486779 (+ 3420us) server_negotiation.cc:407] Connection header received
0928 16:50:25.486901 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:25.486911 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:25.487354 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:25.487565 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:25.489327 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:25.490499 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:25.492013 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:25.493194 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:25.493325 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:25.500420 (+ 7095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:25.500442 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:25.500463 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:25.500525 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:25.518180 (+ 17655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:25.519282 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:25.519293 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:25.519297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:25.519513 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:25.520064 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:25.520069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:25.520072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:25.521204 (+ 1132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:25.524028 (+ 2824us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:25.524235 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 16:50:25.524589 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10491,"thread_start_us":10304,"threads_started":1}
I20260928 16:50:25.526391 16586 ts_manager.cc:194] Registered new tserver with Master: d29db76b68474ab6aa678633dd238fe1 (127.9.93.2:43275)
I20260928 16:50:25.526400 16396 ts_manager.cc:194] Registered new tserver with Master: d29db76b68474ab6aa678633dd238fe1 (127.9.93.2:43275)
I20260928 16:50:25.526746 16245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:25.527107 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:25.475030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.2:42181)
0928 16:50:25.492557 (+ 17527us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:25.492580 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:25.492806 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:25.493400 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:25.493404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:25.494410 (+ 1006us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:25.495594 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:25.495642 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:25.499391 (+ 3749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:25.499396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:25.502170 (+ 2774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:25.502193 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:25.503201 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:25.503210 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:25.503583 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:25.504396 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:25.504439 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:25.509443 (+ 5004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:25.520767 (+ 11324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:25.520773 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:25.520777 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:25.521877 (+ 1100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:25.523410 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:25.523415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:25.523419 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:25.523701 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:25.525001 (+ 1300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:25.525007 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:25.525579 (+ 572us) client_negotiation.cc:769] Sending connection context
0928 16:50:25.525738 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 16:50:25.526075 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17371,"mutex_wait_us":49,"thread_start_us":13016,"threads_started":1}
I20260928 16:50:25.527980 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:25.473563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:42181 (local address 127.9.93.62:33733)
0928 16:50:25.486692 (+ 13129us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:25.486698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:25.492677 (+ 5979us) server_negotiation.cc:407] Connection header received
0928 16:50:25.492975 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:25.492981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:25.493089 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:25.493183 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:25.497976 (+ 4793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:25.499181 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:25.502381 (+ 3200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:25.502911 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:25.503044 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:25.509822 (+ 6778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:25.509830 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:25.509835 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:25.509886 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:25.518728 (+ 8842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:25.522158 (+ 3430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:25.522164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:25.522168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:25.522367 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:25.523884 (+ 1517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:25.523890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:25.523893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:25.524424 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:25.524605 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:25.526853 (+ 2248us) server_negotiation.cc:299] Negotiation successful
0928 16:50:25.527682 (+ 829us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13003,"thread_start_us":12891,"threads_started":1}
I20260928 16:50:25.528927 16913 heartbeater.cc:347] Connected to a master server at 127.9.93.62:33733
I20260928 16:50:25.529091 16913 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:25.529649 16913 heartbeater.cc:511] Master 127.9.93.62:33733 requested a full tablet report, sending...
I20260928 16:50:25.530951 15789 ts_manager.cc:194] Re-registered known tserver with Master: d29db76b68474ab6aa678633dd238fe1 (127.9.93.2:43275)
I20260928 16:50:25.531976 15789 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.93.2@KRBTEST.COM'} at 127.9.93.2:42181
I20260928 16:50:25.617349 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 16121
W20260928 16:50:25.631465 15776 connection.cc:535] server connection from 127.9.93.3:37495 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 16:50:25.632256 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:42893
--local_ip_for_outbound_sockets=127.9.93.3
--tserver_master_addrs=127.9.93.62:33733
--webserver_port=43127
--webserver_interface=127.9.93.3
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.3.keytab
--principal=oryx/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:39869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.93.62:33733,127.9.93.61:38393
--tserver_master_addrs=127.9.93.62:33733,127.9.93.61:38393,127.9.93.60:41929 with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:50:25.843061 16766 heartbeater.cc:503] Master 127.9.93.62:33733 was elected leader, sending a full tablet report...
W20260928 16:50:26.024727 16925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:26.025079 16925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:26.025254 16925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:26.033741 16925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:26.033918 16925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:26.034096 16925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:50:26.045531 16925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.3.keytab
--principal=oryx/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:42893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=43127
--tserver_master_addrs=127.9.93.62:33733,127.9.93.61:38393,127.9.93.60:41929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:26.047436 16925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:26.049050 16925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:26.065205 16931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:26.065229 16933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:26.065166 16930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:26.066118 16925 server_base.cc:1034] running on GCE node
I20260928 16:50:26.066879 16925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:26.068149 16925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:26.069391 16925 hybrid_clock.cc:648] HybridClock initialized: now 1790614226069323 us; error 54 us; skew 500 ppm
I20260928 16:50:26.069787 16925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:26 dist-test-slave-wzcb krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614226, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:26.073614 16925 init.cc:377] Logged in from keytab as oryx/127.9.93.3@KRBTEST.COM (short username oryx)
I20260928 16:50:26.075598 16925 webserver.cc:460] Webserver started at http://127.9.93.3:43127/ using document root <none> and password file <none>
I20260928 16:50:26.076424 16925 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:26.076583 16925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:26.083194 16925 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:50:26.086694 16940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:26.088692 16925 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:50:26.088889 16925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "7397325a370249b5a66a0177c1e2fa13"
format_stamp: "Formatted at 2026-09-28 16:50:22 on dist-test-slave-wzcb"
server_key: "6d1e3bafcfc0b7d553b6b90021fbd017"
server_key_iv: "3b362487759afebc6f0490600a023fe6"
server_key_version: "encryptionkey@0"
I20260928 16:50:26.090140 16925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:26.130857 16925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:26.136001 16925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:26.136337 16925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:26.137550 16925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:50:26.139318 16925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:26.139438 16925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 16:50:26.139571 16925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:26.139652 16925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:26.175844 16925 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:42893
I20260928 16:50:26.175894 17052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:42893 every 8 connection(s)
I20260928 16:50:26.177597 16925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:50:26.179270 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 16925
Sep 28 16:50:26 dist-test-slave-wzcb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614226, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.61@KRBTEST.COM
Sep 28 16:50:26 dist-test-slave-wzcb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614226, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM
Sep 28 16:50:26 dist-test-slave-wzcb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614226, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.60@KRBTEST.COM
I20260928 16:50:26.236709 17058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:26.181115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:46099 (local address 127.9.93.62:33733)
0928 16:50:26.197277 (+ 16162us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:26.197285 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:26.198579 (+ 1294us) server_negotiation.cc:407] Connection header received
0928 16:50:26.207564 (+ 8985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:26.207571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:26.207671 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:26.207788 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:26.211006 (+ 3218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.212182 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.214014 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.214648 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.214773 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:26.224246 (+ 9473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:26.224253 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:26.224257 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:26.224303 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:26.228325 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:26.233259 (+ 4934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:26.233265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:26.233268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:26.233447 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:26.234110 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:26.234115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:26.234122 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:26.234540 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:26.234690 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:26.236049 (+ 1359us) server_negotiation.cc:299] Negotiation successful
0928 16:50:26.236323 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":16001,"thread_start_us":16511,"threads_started":1}
I20260928 16:50:26.238266 17062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:26.181841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.3:46099)
0928 16:50:26.198005 (+ 16164us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:26.198070 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:26.205703 (+ 7633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:26.209884 (+ 4181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:26.209889 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:26.210581 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:26.210813 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:26.210823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.212318 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.212324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:26.213846 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:26.213865 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.215548 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.215555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:26.215731 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:26.220063 (+ 4332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:26.220096 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:26.223881 (+ 3785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:26.230539 (+ 6658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:26.230555 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:26.230582 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:26.233110 (+ 2528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:26.233782 (+ 672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:26.233786 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:26.233789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:26.233962 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:26.234840 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:26.234851 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:26.235473 (+ 622us) client_negotiation.cc:769] Sending connection context
0928 16:50:26.235897 (+ 424us) client_negotiation.cc:240] Negotiation successful
0928 16:50:26.236559 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15981,"mutex_wait_us":383,"thread_start_us":15173,"threads_started":1}
I20260928 16:50:26.238844 17060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:26.181841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:38393 (local address 127.9.93.3:59429)
0928 16:50:26.197555 (+ 15714us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:26.198374 (+ 819us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:26.201542 (+ 3168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:26.203024 (+ 1482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:26.203053 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:26.204193 (+ 1140us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:26.205353 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:26.205390 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.207480 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.207487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:26.209648 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:26.209669 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.214415 (+ 4746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.214422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:26.214800 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:26.217987 (+ 3187us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:26.218034 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:26.223880 (+ 5846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:26.231700 (+ 7820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:26.231706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:26.231709 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:26.232530 (+ 821us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:26.233278 (+ 748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:26.233283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:26.233287 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:26.233631 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:26.237599 (+ 3968us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:26.237604 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:26.237799 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 16:50:26.237942 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 16:50:26.238360 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15278,"thread_start_us":14965,"threads_started":1}
I20260928 16:50:26.240188 17054 heartbeater.cc:347] Connected to a master server at 127.9.93.62:33733
I20260928 16:50:26.240530 17054 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:26.241492 17054 heartbeater.cc:511] Master 127.9.93.62:33733 requested a full tablet report, sending...
I20260928 16:50:26.243427 17059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:26.181159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:59429 (local address 127.9.93.61:38393)
0928 16:50:26.192009 (+ 10850us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:26.192018 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:26.198500 (+ 6482us) server_negotiation.cc:407] Connection header received
0928 16:50:26.202083 (+ 3583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:26.202092 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:26.202225 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:26.202375 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:26.205670 (+ 3295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.207242 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.213460 (+ 6218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.214149 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.214317 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:26.224242 (+ 9925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:26.224250 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:26.224255 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:26.224307 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:26.231567 (+ 7260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:26.232800 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:26.232806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:26.232810 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:26.233001 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:26.233780 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:26.233786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:26.233790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:26.234908 (+ 1118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:26.235058 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:26.238344 (+ 3286us) server_negotiation.cc:299] Negotiation successful
0928 16:50:26.239863 (+ 1519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":10671,"thread_start_us":10536,"threads_started":1}
I20260928 16:50:26.243716 15789 ts_manager.cc:194] Re-registered known tserver with Master: 7397325a370249b5a66a0177c1e2fa13 (127.9.93.3:42893)
I20260928 16:50:26.244976 15789 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.93.3@KRBTEST.COM'} at 127.9.93.3:46099
I20260928 16:50:26.245962 17055 heartbeater.cc:347] Connected to a master server at 127.9.93.61:38393
I20260928 16:50:26.246150 17055 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:26.246717 17055 heartbeater.cc:511] Master 127.9.93.61:38393 requested a full tablet report, sending...
I20260928 16:50:26.247505 17072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:26.211218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52444 (local address 127.9.93.61:38393)
0928 16:50:26.213970 (+ 2752us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:26.213978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:26.213997 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:50:26.214056 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:26.214062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:26.214143 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:26.214254 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:26.220662 (+ 6408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.222003 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.224667 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.225397 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.225530 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:26.229851 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:26.229859 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:26.229864 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:26.229923 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:26.237380 (+ 7457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:26.238666 (+ 1286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:26.238672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:26.238676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:26.238863 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:26.244810 (+ 5947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:26.244816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:26.244819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:26.245242 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:26.245568 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:26.246696 (+ 1128us) server_negotiation.cc:299] Negotiation successful
0928 16:50:26.246992 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2605,"thread_start_us":2787,"threads_started":1}
I20260928 16:50:26.248337 16396 ts_manager.cc:194] Registered new tserver with Master: 7397325a370249b5a66a0177c1e2fa13 (127.9.93.3:42893)
I20260928 16:50:26.248778 17065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:26.198815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41929 (local address 127.9.93.3:51491)
0928 16:50:26.214176 (+ 15361us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:26.214198 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:26.224496 (+ 10298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:26.225489 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:26.225494 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:26.227767 (+ 2273us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:26.228155 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:26.228170 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.229850 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.229856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:26.231467 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:26.231484 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.232735 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.232743 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:26.232966 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:26.234362 (+ 1396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:26.234385 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:26.239913 (+ 5528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:26.245099 (+ 5186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:26.245105 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:26.245109 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:26.245823 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:26.246371 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:26.246376 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:26.246380 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:26.246807 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:26.247617 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:26.247621 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:26.247784 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 16:50:26.247927 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 16:50:26.248311 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8416,"thread_start_us":8131,"threads_started":1}
I20260928 16:50:26.248862 17067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:26.204370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32846 (local address 127.9.93.62:33733)
0928 16:50:26.206774 (+ 2404us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:26.206780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:26.211735 (+ 4955us) server_negotiation.cc:407] Connection header received
0928 16:50:26.211801 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:26.211807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:26.211882 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:26.211980 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:26.216246 (+ 4266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.217438 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.221378 (+ 3940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.222134 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.222272 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:26.231856 (+ 9584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:26.231862 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:26.231867 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:26.231914 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:26.236269 (+ 4355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:26.238341 (+ 2072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:26.238348 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:26.238352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:26.238507 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:26.241224 (+ 2717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:26.241230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:26.241233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:26.241640 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:26.241878 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:26.244653 (+ 2775us) server_negotiation.cc:299] Negotiation successful
0928 16:50:26.244949 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":2267,"thread_start_us":2989,"threads_started":1}
I20260928 16:50:26.250053 17071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:26.209963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51926 (local address 127.9.93.60:41929)
0928 16:50:26.220658 (+ 10695us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:26.220666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:26.221398 (+ 732us) server_negotiation.cc:407] Connection header received
0928 16:50:26.221715 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:26.221721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:26.221818 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:26.221917 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:26.226267 (+ 4350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.227540 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.230747 (+ 3207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.231456 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.231568 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:26.241740 (+ 10172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:26.241749 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:26.241758 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:26.241807 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:26.246492 (+ 4685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:26.247473 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:26.247479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:26.247482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:26.247668 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:26.248150 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:26.248155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:26.248158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:26.248509 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:26.248766 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:26.249393 (+ 627us) server_negotiation.cc:299] Negotiation successful
0928 16:50:26.249661 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10533,"thread_start_us":10278,"threads_started":1}
I20260928 16:50:26.250130 17066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:26.198840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:51491 (local address 127.9.93.60:41929)
0928 16:50:26.206416 (+ 7576us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:26.206424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:26.224443 (+ 18019us) server_negotiation.cc:407] Connection header received
0928 16:50:26.224621 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:26.224627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:26.224720 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:26.224891 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:26.228393 (+ 3502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.229628 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.231637 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.232306 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.232434 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:26.240239 (+ 7805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:26.240248 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:26.240253 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:26.240307 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:26.244788 (+ 4481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:26.245988 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:26.245995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:26.245998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:26.246197 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:26.246947 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:26.246952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:26.246955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:26.247371 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:26.247518 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:26.249139 (+ 1621us) server_negotiation.cc:299] Negotiation successful
0928 16:50:26.249398 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7391,"thread_start_us":8065,"threads_started":1}
I20260928 16:50:26.251394 17053 heartbeater.cc:347] Connected to a master server at 127.9.93.60:41929
I20260928 16:50:26.251577 17053 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:26.252161 17053 heartbeater.cc:511] Master 127.9.93.60:41929 requested a full tablet report, sending...
I20260928 16:50:26.253453 16586 ts_manager.cc:194] Registered new tserver with Master: 7397325a370249b5a66a0177c1e2fa13 (127.9.93.3:42893)
I20260928 16:50:26.255362 15789 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:50:26.263375 15789 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:50:26.296635 17084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:26.284631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52596 (local address 127.9.93.1:40577)
0928 16:50:26.287177 (+ 2546us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:26.287189 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:26.287228 (+ 39us) server_negotiation.cc:407] Connection header received
0928 16:50:26.287394 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:26.287405 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:26.287768 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:26.288601 (+ 833us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:50:26.289210 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.290700 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.293851 (+ 3151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.295432 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.295609 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:26.295871 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:26.296121 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 16:50:26.296309 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2312,"thread_start_us":2161,"threads_started":1}
I20260928 16:50:26.296689 17083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:26.284288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:40577 (local address 127.0.0.1:52596)
0928 16:50:26.284973 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:26.284994 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:26.285366 (+ 372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:26.288022 (+ 2656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:26.288027 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:26.288056 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:50:26.288404 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:26.288414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.290860 (+ 2446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.290865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:26.292934 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:26.292950 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.295699 (+ 2749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.295709 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:26.295900 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:26.295939 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 16:50:26.296032 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 16:50:26.296238 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":1,"thread_start_us":1449,"threads_started":1}
I20260928 16:50:26.298970 17086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:26.285540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54746 (local address 127.9.93.2:43275)
0928 16:50:26.288156 (+ 2616us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:26.288166 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:26.288200 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:50:26.288327 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:26.288338 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:26.288676 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:26.288921 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:50:26.289996 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.291490 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.294881 (+ 3391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.296504 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.296650 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:26.296994 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:26.298430 (+ 1436us) server_negotiation.cc:299] Negotiation successful
0928 16:50:26.298663 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2432,"thread_start_us":2278,"threads_started":1}
I20260928 16:50:26.299165 17088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:26.286301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:42893 (local address 127.0.0.1:37802)
0928 16:50:26.289704 (+ 3403us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:26.289722 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:26.289851 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:26.290928 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:26.290933 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:26.290958 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:50:26.291496 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:26.291507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.293792 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.293798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:26.296275 (+ 2477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:26.296291 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.298373 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.298380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:26.298576 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:26.298610 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 16:50:26.298680 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 16:50:26.298826 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3277,"thread_start_us":3091,"threads_started":1}
I20260928 16:50:26.300078 17087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:26.285408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37802 (local address 127.9.93.3:42893)
0928 16:50:26.287657 (+ 2249us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:26.287668 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:26.290268 (+ 2600us) server_negotiation.cc:407] Connection header received
0928 16:50:26.290388 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:26.290396 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:26.290670 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:26.290879 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:50:26.291745 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.293617 (+ 1872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.296501 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.298039 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.299345 (+ 1306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:26.299586 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:26.299653 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 16:50:26.299806 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2040,"thread_start_us":1811,"threads_started":1}
I20260928 16:50:26.301510 16700 tablet_service.cc:1467] Processing CreateTablet for tablet 7bc0f91a09a54ebeb0146190414d2abe (DEFAULT_TABLE table=test-table [id=0f65b6d242be4f50990e7ded1cd2eabc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:50:26.303505 16700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bc0f91a09a54ebeb0146190414d2abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:26.303985 17085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:26.285119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:43275 (local address 127.0.0.1:54746)
0928 16:50:26.286004 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:26.286026 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:26.286155 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:26.288898 (+ 2743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:26.288904 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:26.288930 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:50:26.289442 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:26.289461 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.291697 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.291703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:26.294645 (+ 2942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:26.294660 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.297375 (+ 2715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.297382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:26.297552 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:26.297585 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 16:50:26.297681 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 16:50:26.297894 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":1,"thread_start_us":614,"threads_started":1}
I20260928 16:50:26.304538 16988 tablet_service.cc:1467] Processing CreateTablet for tablet 7bc0f91a09a54ebeb0146190414d2abe (DEFAULT_TABLE table=test-table [id=0f65b6d242be4f50990e7ded1cd2eabc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:50:26.306311 16988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bc0f91a09a54ebeb0146190414d2abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:26.308785 16847 tablet_service.cc:1467] Processing CreateTablet for tablet 7bc0f91a09a54ebeb0146190414d2abe (DEFAULT_TABLE table=test-table [id=0f65b6d242be4f50990e7ded1cd2eabc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:50:26.310436 16847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bc0f91a09a54ebeb0146190414d2abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:26.320519 17089 tablet_bootstrap.cc:492] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b: Bootstrap starting.
I20260928 16:50:26.327207 17089 tablet_bootstrap.cc:654] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:26.329435 17089 log.cc:824] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b: Log is configured to *not* fsync() on all Append() calls
I20260928 16:50:26.331209 17090 tablet_bootstrap.cc:492] T 7bc0f91a09a54ebeb0146190414d2abe P 7397325a370249b5a66a0177c1e2fa13: Bootstrap starting.
I20260928 16:50:26.334674 17091 tablet_bootstrap.cc:492] T 7bc0f91a09a54ebeb0146190414d2abe P d29db76b68474ab6aa678633dd238fe1: Bootstrap starting.
I20260928 16:50:26.335759 17090 tablet_bootstrap.cc:654] T 7bc0f91a09a54ebeb0146190414d2abe P 7397325a370249b5a66a0177c1e2fa13: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:26.337317 17090 log.cc:824] T 7bc0f91a09a54ebeb0146190414d2abe P 7397325a370249b5a66a0177c1e2fa13: Log is configured to *not* fsync() on all Append() calls
I20260928 16:50:26.340464 17091 tablet_bootstrap.cc:654] T 7bc0f91a09a54ebeb0146190414d2abe P d29db76b68474ab6aa678633dd238fe1: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:26.342084 17091 log.cc:824] T 7bc0f91a09a54ebeb0146190414d2abe P d29db76b68474ab6aa678633dd238fe1: Log is configured to *not* fsync() on all Append() calls
I20260928 16:50:26.342919 17089 tablet_bootstrap.cc:492] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b: No bootstrap required, opened a new log
I20260928 16:50:26.343325 17089 ts_tablet_manager.cc:1397] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b: Time spent bootstrapping tablet: real 0.023s user 0.015s sys 0.000s
I20260928 16:50:26.344537 17090 tablet_bootstrap.cc:492] T 7bc0f91a09a54ebeb0146190414d2abe P 7397325a370249b5a66a0177c1e2fa13: No bootstrap required, opened a new log
I20260928 16:50:26.344935 17090 ts_tablet_manager.cc:1397] T 7bc0f91a09a54ebeb0146190414d2abe P 7397325a370249b5a66a0177c1e2fa13: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.004s
I20260928 16:50:26.346709 17091 tablet_bootstrap.cc:492] T 7bc0f91a09a54ebeb0146190414d2abe P d29db76b68474ab6aa678633dd238fe1: No bootstrap required, opened a new log
I20260928 16:50:26.347009 17091 ts_tablet_manager.cc:1397] T 7bc0f91a09a54ebeb0146190414d2abe P d29db76b68474ab6aa678633dd238fe1: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20260928 16:50:26.355144 17091 raft_consensus.cc:348] T 7bc0f91a09a54ebeb0146190414d2abe P d29db76b68474ab6aa678633dd238fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5622e16bb5462793db42b715d8d31b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40577 } } peers { permanent_uuid: "7397325a370249b5a66a0177c1e2fa13" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42893 } } peers { permanent_uuid: "d29db76b68474ab6aa678633dd238fe1" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43275 } }
I20260928 16:50:26.355152 17089 raft_consensus.cc:348] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5622e16bb5462793db42b715d8d31b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40577 } } peers { permanent_uuid: "7397325a370249b5a66a0177c1e2fa13" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42893 } } peers { permanent_uuid: "d29db76b68474ab6aa678633dd238fe1" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43275 } }
I20260928 16:50:26.355705 17089 raft_consensus.cc:374] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:26.355691 17091 raft_consensus.cc:374] T 7bc0f91a09a54ebeb0146190414d2abe P d29db76b68474ab6aa678633dd238fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:26.355825 17089 raft_consensus.cc:729] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c5622e16bb5462793db42b715d8d31b, State: Initialized, Role: FOLLOWER
I20260928 16:50:26.355868 17091 raft_consensus.cc:729] T 7bc0f91a09a54ebeb0146190414d2abe P d29db76b68474ab6aa678633dd238fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d29db76b68474ab6aa678633dd238fe1, State: Initialized, Role: FOLLOWER
I20260928 16:50:26.356621 17091 consensus_queue.cc:260] T 7bc0f91a09a54ebeb0146190414d2abe P d29db76b68474ab6aa678633dd238fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5622e16bb5462793db42b715d8d31b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40577 } } peers { permanent_uuid: "7397325a370249b5a66a0177c1e2fa13" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42893 } } peers { permanent_uuid: "d29db76b68474ab6aa678633dd238fe1" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43275 } }
I20260928 16:50:26.356650 17089 consensus_queue.cc:260] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5622e16bb5462793db42b715d8d31b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40577 } } peers { permanent_uuid: "7397325a370249b5a66a0177c1e2fa13" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42893 } } peers { permanent_uuid: "d29db76b68474ab6aa678633dd238fe1" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43275 } }
I20260928 16:50:26.356952 17090 raft_consensus.cc:348] T 7bc0f91a09a54ebeb0146190414d2abe P 7397325a370249b5a66a0177c1e2fa13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5622e16bb5462793db42b715d8d31b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40577 } } peers { permanent_uuid: "7397325a370249b5a66a0177c1e2fa13" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42893 } } peers { permanent_uuid: "d29db76b68474ab6aa678633dd238fe1" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43275 } }
I20260928 16:50:26.357620 17090 raft_consensus.cc:374] T 7bc0f91a09a54ebeb0146190414d2abe P 7397325a370249b5a66a0177c1e2fa13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:26.357733 17090 raft_consensus.cc:729] T 7bc0f91a09a54ebeb0146190414d2abe P 7397325a370249b5a66a0177c1e2fa13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7397325a370249b5a66a0177c1e2fa13, State: Initialized, Role: FOLLOWER
I20260928 16:50:26.358675 17090 consensus_queue.cc:260] T 7bc0f91a09a54ebeb0146190414d2abe P 7397325a370249b5a66a0177c1e2fa13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5622e16bb5462793db42b715d8d31b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40577 } } peers { permanent_uuid: "7397325a370249b5a66a0177c1e2fa13" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42893 } } peers { permanent_uuid: "d29db76b68474ab6aa678633dd238fe1" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43275 } }
I20260928 16:50:26.360292 17054 heartbeater.cc:503] Master 127.9.93.62:33733 was elected leader, sending a full tablet report...
I20260928 16:50:26.362529 17091 ts_tablet_manager.cc:1428] T 7bc0f91a09a54ebeb0146190414d2abe P d29db76b68474ab6aa678633dd238fe1: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
I20260928 16:50:26.363524 16913 heartbeater.cc:503] Master 127.9.93.62:33733 was elected leader, sending a full tablet report...
I20260928 16:50:26.364918 17089 ts_tablet_manager.cc:1428] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b: Time spent starting tablet: real 0.021s user 0.014s sys 0.004s
I20260928 16:50:26.369863 17090 ts_tablet_manager.cc:1428] T 7bc0f91a09a54ebeb0146190414d2abe P 7397325a370249b5a66a0177c1e2fa13: Time spent starting tablet: real 0.025s user 0.013s sys 0.002s
W20260928 16:50:26.430356 17056 tablet.cc:2369] T 7bc0f91a09a54ebeb0146190414d2abe P 7397325a370249b5a66a0177c1e2fa13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:50:26.442351 17096 raft_consensus.cc:482] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:50:26.442718 17096 raft_consensus.cc:504] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5622e16bb5462793db42b715d8d31b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40577 } } peers { permanent_uuid: "7397325a370249b5a66a0177c1e2fa13" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42893 } } peers { permanent_uuid: "d29db76b68474ab6aa678633dd238fe1" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43275 } }
I20260928 16:50:26.444834 17096 leader_election.cc:288] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7397325a370249b5a66a0177c1e2fa13 (127.9.93.3:42893), d29db76b68474ab6aa678633dd238fe1 (127.9.93.2:43275)
I20260928 16:50:26.454068 17086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:26.445454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:33947 (local address 127.9.93.2:43275)
0928 16:50:26.445664 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:26.445670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:26.446486 (+ 816us) server_negotiation.cc:407] Connection header received
0928 16:50:26.446730 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:26.446736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:26.446829 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:26.446928 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:50:26.447783 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.448981 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.451613 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.452949 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.453098 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:26.453187 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:26.453627 (+ 440us) server_negotiation.cc:299] Negotiation successful
0928 16:50:26.453778 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 16:50:26.454064 17087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:26.445394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:41353 (local address 127.9.93.3:42893)
0928 16:50:26.445662 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:26.445670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:26.446485 (+ 815us) server_negotiation.cc:407] Connection header received
0928 16:50:26.446730 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:26.446736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:26.446829 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:26.446928 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:50:26.447784 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.448960 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.451465 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.452756 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.452928 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:26.453056 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:26.453554 (+ 498us) server_negotiation.cc:299] Negotiation successful
0928 16:50:26.453733 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 16:50:26.454252 17099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:26.445308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:43275 (local address 127.9.93.1:33947)
0928 16:50:26.446348 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:26.446362 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:26.446551 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:26.446954 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:26.446959 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:26.446996 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:50:26.447549 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:26.447566 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.449195 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.449199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:26.451416 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:26.451434 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.453221 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.453228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:26.453410 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:26.453447 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 16:50:26.453525 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 16:50:26.453739 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"mutex_wait_us":76,"spinlock_wait_cycles":16768,"thread_start_us":659,"threads_started":1}
I20260928 16:50:26.454321 17098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:26.445146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:42893 (local address 127.9.93.1:41353)
0928 16:50:26.446315 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:26.446343 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:26.446551 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:26.446954 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:26.446959 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:26.446996 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:50:26.447648 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:26.447661 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.449148 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.449154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:26.451231 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:26.451253 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.453095 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.453104 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:26.453340 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:26.453386 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 16:50:26.453509 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 16:50:26.453742 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"mutex_wait_us":141,"spinlock_wait_cycles":17280,"thread_start_us":735,"threads_started":1}
I20260928 16:50:26.455623 16867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bc0f91a09a54ebeb0146190414d2abe" candidate_uuid: "8c5622e16bb5462793db42b715d8d31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d29db76b68474ab6aa678633dd238fe1" is_pre_election: true
I20260928 16:50:26.455624 17008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bc0f91a09a54ebeb0146190414d2abe" candidate_uuid: "8c5622e16bb5462793db42b715d8d31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7397325a370249b5a66a0177c1e2fa13" is_pre_election: true
I20260928 16:50:26.456224 16867 raft_consensus.cc:2454] T 7bc0f91a09a54ebeb0146190414d2abe P d29db76b68474ab6aa678633dd238fe1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c5622e16bb5462793db42b715d8d31b in term 0.
I20260928 16:50:26.456223 17008 raft_consensus.cc:2454] T 7bc0f91a09a54ebeb0146190414d2abe P 7397325a370249b5a66a0177c1e2fa13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c5622e16bb5462793db42b715d8d31b in term 0.
I20260928 16:50:26.457051 16655 leader_election.cc:302] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7397325a370249b5a66a0177c1e2fa13, 8c5622e16bb5462793db42b715d8d31b; no voters:
I20260928 16:50:26.457504 17096 raft_consensus.cc:2781] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:50:26.457780 17096 raft_consensus.cc:482] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:50:26.457878 17096 raft_consensus.cc:3037] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:26.460415 17096 raft_consensus.cc:504] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5622e16bb5462793db42b715d8d31b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40577 } } peers { permanent_uuid: "7397325a370249b5a66a0177c1e2fa13" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42893 } } peers { permanent_uuid: "d29db76b68474ab6aa678633dd238fe1" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43275 } }
I20260928 16:50:26.461161 17096 leader_election.cc:288] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b [CANDIDATE]: Term 1 election: Requested vote from peers 7397325a370249b5a66a0177c1e2fa13 (127.9.93.3:42893), d29db76b68474ab6aa678633dd238fe1 (127.9.93.2:43275)
I20260928 16:50:26.461812 17008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bc0f91a09a54ebeb0146190414d2abe" candidate_uuid: "8c5622e16bb5462793db42b715d8d31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7397325a370249b5a66a0177c1e2fa13"
I20260928 16:50:26.461858 16867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bc0f91a09a54ebeb0146190414d2abe" candidate_uuid: "8c5622e16bb5462793db42b715d8d31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d29db76b68474ab6aa678633dd238fe1"
I20260928 16:50:26.462097 16867 raft_consensus.cc:3037] T 7bc0f91a09a54ebeb0146190414d2abe P d29db76b68474ab6aa678633dd238fe1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:26.462110 17008 raft_consensus.cc:3037] T 7bc0f91a09a54ebeb0146190414d2abe P 7397325a370249b5a66a0177c1e2fa13 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:26.465871 17008 raft_consensus.cc:2454] T 7bc0f91a09a54ebeb0146190414d2abe P 7397325a370249b5a66a0177c1e2fa13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c5622e16bb5462793db42b715d8d31b in term 1.
I20260928 16:50:26.465871 16867 raft_consensus.cc:2454] T 7bc0f91a09a54ebeb0146190414d2abe P d29db76b68474ab6aa678633dd238fe1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c5622e16bb5462793db42b715d8d31b in term 1.
I20260928 16:50:26.466518 16654 leader_election.cc:302] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c5622e16bb5462793db42b715d8d31b, d29db76b68474ab6aa678633dd238fe1; no voters:
I20260928 16:50:26.466946 17096 raft_consensus.cc:2781] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:50:26.467547 17096 raft_consensus.cc:686] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b [term 1 LEADER]: Becoming Leader. State: Replica: 8c5622e16bb5462793db42b715d8d31b, State: Running, Role: LEADER
I20260928 16:50:26.468191 17096 consensus_queue.cc:237] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5622e16bb5462793db42b715d8d31b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40577 } } peers { permanent_uuid: "7397325a370249b5a66a0177c1e2fa13" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42893 } } peers { permanent_uuid: "d29db76b68474ab6aa678633dd238fe1" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43275 } }
I20260928 16:50:26.473610 15787 catalog_manager.cc:5495] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c5622e16bb5462793db42b715d8d31b (127.9.93.1). New cstate: current_term: 1 leader_uuid: "8c5622e16bb5462793db42b715d8d31b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5622e16bb5462793db42b715d8d31b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7397325a370249b5a66a0177c1e2fa13" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d29db76b68474ab6aa678633dd238fe1" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43275 } health_report { overall_health: UNKNOWN } } }
W20260928 16:50:26.474977 16915 tablet.cc:2369] T 7bc0f91a09a54ebeb0146190414d2abe P d29db76b68474ab6aa678633dd238fe1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:50:26.501196 17084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:26.495206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52612 (local address 127.9.93.1:40577)
0928 16:50:26.495463 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:26.495469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:26.495550 (+ 81us) server_negotiation.cc:407] Connection header received
0928 16:50:26.495871 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:26.495875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:26.495967 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:26.496120 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:50:26.496530 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.497287 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.498762 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.499220 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.499370 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:26.499936 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:50:26.500534 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:50:26.500702 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:26.500784 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 16:50:26.500891 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 16:50:26.511188 16867 raft_consensus.cc:1261] T 7bc0f91a09a54ebeb0146190414d2abe P d29db76b68474ab6aa678633dd238fe1 [term 1 FOLLOWER]: Refusing update from remote peer 8c5622e16bb5462793db42b715d8d31b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:50:26.511191 17008 raft_consensus.cc:1261] T 7bc0f91a09a54ebeb0146190414d2abe P 7397325a370249b5a66a0177c1e2fa13 [term 1 FOLLOWER]: Refusing update from remote peer 8c5622e16bb5462793db42b715d8d31b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:50:26.512482 17100 consensus_queue.cc:1035] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7397325a370249b5a66a0177c1e2fa13" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:50:26.512971 17096 consensus_queue.cc:1035] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d29db76b68474ab6aa678633dd238fe1" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:50:26.529685 17104 mvcc.cc:205] Tried to move back new op lower bound from 7334355871777751040 to 7334355871617961984. Current Snapshot: MvccSnapshot[applied={T|T < 7334355871777751040}]
I20260928 16:50:26.552203 15787 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:50:26.552569 15787 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:50:26.560077 15787 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2577aff545514d4389450abad8337d7b: Sending DeleteTablet for 3 replicas of tablet 7bc0f91a09a54ebeb0146190414d2abe
I20260928 16:50:26.561764 16700 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bc0f91a09a54ebeb0146190414d2abe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:26 UTC) from {username='oryx', principal='oryx/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:52596
I20260928 16:50:26.562702 16988 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bc0f91a09a54ebeb0146190414d2abe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:26 UTC) from {username='oryx', principal='oryx/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:37802
I20260928 16:50:26.563611 16847 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bc0f91a09a54ebeb0146190414d2abe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:26 UTC) from {username='oryx', principal='oryx/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:54746
I20260928 16:50:26.563779 17115 tablet_replica.cc:330] T 7bc0f91a09a54ebeb0146190414d2abe P 7397325a370249b5a66a0177c1e2fa13: stopping tablet replica
I20260928 16:50:26.564311 17115 raft_consensus.cc:2229] T 7bc0f91a09a54ebeb0146190414d2abe P 7397325a370249b5a66a0177c1e2fa13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:50:26.564461 17114 tablet_replica.cc:330] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b: stopping tablet replica
I20260928 16:50:26.564927 17115 raft_consensus.cc:2258] T 7bc0f91a09a54ebeb0146190414d2abe P 7397325a370249b5a66a0177c1e2fa13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:26.564954 17114 raft_consensus.cc:2229] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:50:26.565682 17114 raft_consensus.cc:2258] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:26.567847 17115 ts_tablet_manager.cc:1905] T 7bc0f91a09a54ebeb0146190414d2abe P 7397325a370249b5a66a0177c1e2fa13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:50:26.568100 17114 ts_tablet_manager.cc:1905] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:50:26.576519 17114 ts_tablet_manager.cc:1918] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:50:26.576786 17114 log.cc:1196] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/wal/wals/7bc0f91a09a54ebeb0146190414d2abe
I20260928 16:50:26.576860 17115 ts_tablet_manager.cc:1918] T 7bc0f91a09a54ebeb0146190414d2abe P 7397325a370249b5a66a0177c1e2fa13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:50:26.577131 17115 log.cc:1196] T 7bc0f91a09a54ebeb0146190414d2abe P 7397325a370249b5a66a0177c1e2fa13: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/7bc0f91a09a54ebeb0146190414d2abe
I20260928 16:50:26.577479 17114 ts_tablet_manager.cc:1939] T 7bc0f91a09a54ebeb0146190414d2abe P 8c5622e16bb5462793db42b715d8d31b: Deleting consensus metadata
I20260928 16:50:26.577730 17115 ts_tablet_manager.cc:1939] T 7bc0f91a09a54ebeb0146190414d2abe P 7397325a370249b5a66a0177c1e2fa13: Deleting consensus metadata
I20260928 16:50:26.579809 15776 catalog_manager.cc:4841] TS 8c5622e16bb5462793db42b715d8d31b (127.9.93.1:40577): tablet 7bc0f91a09a54ebeb0146190414d2abe (table test-table [id=0f65b6d242be4f50990e7ded1cd2eabc]) successfully deleted
I20260928 16:50:26.579851 15775 catalog_manager.cc:4841] TS 7397325a370249b5a66a0177c1e2fa13 (127.9.93.3:42893): tablet 7bc0f91a09a54ebeb0146190414d2abe (table test-table [id=0f65b6d242be4f50990e7ded1cd2eabc]) successfully deleted
I20260928 16:50:26.581821 17116 tablet_replica.cc:330] T 7bc0f91a09a54ebeb0146190414d2abe P d29db76b68474ab6aa678633dd238fe1: stopping tablet replica
I20260928 16:50:26.582159 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 15757
I20260928 16:50:26.582442 17116 raft_consensus.cc:2229] T 7bc0f91a09a54ebeb0146190414d2abe P d29db76b68474ab6aa678633dd238fe1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:50:26.583104 17116 raft_consensus.cc:2258] T 7bc0f91a09a54ebeb0146190414d2abe P d29db76b68474ab6aa678633dd238fe1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:26.584443 15820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:26.586453 17116 ts_tablet_manager.cc:1905] T 7bc0f91a09a54ebeb0146190414d2abe P d29db76b68474ab6aa678633dd238fe1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:50:26.595818 17116 ts_tablet_manager.cc:1918] T 7bc0f91a09a54ebeb0146190414d2abe P d29db76b68474ab6aa678633dd238fe1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:50:26.596156 17116 log.cc:1196] T 7bc0f91a09a54ebeb0146190414d2abe P d29db76b68474ab6aa678633dd238fe1: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/7bc0f91a09a54ebeb0146190414d2abe
I20260928 16:50:26.596873 17116 ts_tablet_manager.cc:1939] T 7bc0f91a09a54ebeb0146190414d2abe P d29db76b68474ab6aa678633dd238fe1: Deleting consensus metadata
I20260928 16:50:26.601861 17072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:26.565272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52454 (local address 127.9.93.61:38393)
0928 16:50:26.565489 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:26.565495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:26.581734 (+ 16239us) server_negotiation.cc:407] Connection header received
0928 16:50:26.581837 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:26.581842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:26.581957 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:26.582069 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:26.584902 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.586072 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.588384 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.589076 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.589198 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:26.593284 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:26.593292 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:26.593297 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:26.593353 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:26.597367 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:26.599118 (+ 1751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:26.599126 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:26.599129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:26.599391 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:26.599993 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:26.599999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:26.600002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:26.600454 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:26.600785 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:26.601210 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 16:50:26.601495 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:50:26.621202 17066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:26.581304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51938 (local address 127.9.93.60:41929)
0928 16:50:26.581520 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:26.581527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:26.602421 (+ 20894us) server_negotiation.cc:407] Connection header received
0928 16:50:26.602745 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:26.602752 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:26.602878 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:26.602959 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:26.605180 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.606449 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.608578 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.609262 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.609401 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:26.613282 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:26.613289 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:26.613294 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:26.613358 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:26.617293 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:26.618415 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:26.618421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:26.618425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:26.618626 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:26.619168 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:26.619175 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:26.619179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:26.619707 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:26.620002 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:26.620487 (+ 485us) server_negotiation.cc:299] Negotiation successful
0928 16:50:26.620817 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 16:50:26.730110 15774 catalog_manager.cc:4841] TS d29db76b68474ab6aa678633dd238fe1 (127.9.93.2:43275): tablet 7bc0f91a09a54ebeb0146190414d2abe (table test-table [id=0f65b6d242be4f50990e7ded1cd2eabc]) successfully deleted
W20260928 16:50:26.730382 15774 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.9.93.2:43275, user_credentials={real_user=slave}} blocked reactor thread for 130950us
I20260928 16:50:26.730968 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 15757
I20260928 16:50:26.784090 17072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:26.765223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52458 (local address 127.9.93.61:38393)
0928 16:50:26.765461 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:26.765470 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:26.766687 (+ 1217us) server_negotiation.cc:407] Connection header received
0928 16:50:26.767027 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:26.767034 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:26.767158 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:26.767281 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:26.769451 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.770711 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.772574 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.773058 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.773193 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:26.777095 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:26.777104 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:26.777110 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:26.777170 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:26.780537 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:26.781510 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:26.781517 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:26.781520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:26.781695 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:26.782126 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:26.782131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:26.782133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:26.782572 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:26.782886 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:26.783266 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 16:50:26.783593 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 16:50:26.786362 17066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:26.766871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51940 (local address 127.9.93.60:41929)
0928 16:50:26.767072 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:26.767077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:26.767288 (+ 211us) server_negotiation.cc:407] Connection header received
0928 16:50:26.767626 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:26.767632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:26.767721 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:26.767824 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:26.769696 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.770465 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.772504 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:26.773230 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:26.773364 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:26.778096 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:26.778104 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:26.778110 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:26.778163 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:26.782062 (+ 3899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:26.782895 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:26.782902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:26.782905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:26.783060 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:26.783633 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:26.783639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:26.783642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:26.784048 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:26.784654 (+ 606us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:26.785866 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0928 16:50:26.786096 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 16:50:27.541702 17125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:27.540794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.1:55413)
0928 16:50:27.541315 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:27.541455 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":252,"threads_started":1}
W20260928 16:50:27.542421 16766 heartbeater.cc:644] Failed to heartbeat to 127.9.93.62:33733 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
I20260928 16:50:27.546962 17126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:27.545690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.2:45867)
0928 16:50:27.546554 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:27.546738 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":724,"threads_started":1}
W20260928 16:50:27.548190 16913 heartbeater.cc:644] Failed to heartbeat to 127.9.93.62:33733 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
I20260928 16:50:27.548888 17127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:27.546985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.3:33653)
0928 16:50:27.548456 (+ 1471us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:27.548631 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1331,"thread_start_us":1220,"threads_started":1}
W20260928 16:50:27.549412 17054 heartbeater.cc:644] Failed to heartbeat to 127.9.93.62:33733 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
I20260928 16:50:28.061288 17128 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2577aff545514d4389450abad8337d7b)
I20260928 16:50:28.061587 17128 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: false } }
I20260928 16:50:28.062427 17129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2577aff545514d4389450abad8337d7b)
I20260928 16:50:28.062639 17128 leader_election.cc:288] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2577aff545514d4389450abad8337d7b (127.9.93.62:33733), 31a246c9c6804fdb8430171f57199a93 (127.9.93.61:38393)
I20260928 16:50:28.062754 17129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: false } }
I20260928 16:50:28.073215 17129 leader_election.cc:288] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2577aff545514d4389450abad8337d7b (127.9.93.62:33733), 6cebda140f974905b3e3480b980f7a2d (127.9.93.60:41929)
I20260928 16:50:28.073395 17135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:28.065032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.0.0.1:33236)
0928 16:50:28.072839 (+ 7807us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:28.073104 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7663,"threads_started":1}
I20260928 16:50:28.079077 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:28.062893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.0.0.1:33224)
0928 16:50:28.078475 (+ 15582us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:28.078721 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15067,"thread_start_us":15276,"threads_started":1}
W20260928 16:50:28.079499 16570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
W20260928 16:50:28.080339 16380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
I20260928 16:50:28.083729 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:28.071851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.9.93.60:41929)
0928 16:50:28.072680 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:28.072688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:28.073740 (+ 1052us) server_negotiation.cc:407] Connection header received
0928 16:50:28.074004 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:28.074012 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:28.074169 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:28.074313 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:50:28.076241 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:28.077511 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:28.080232 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:28.081757 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:28.081925 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:28.082039 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:28.083116 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0928 16:50:28.083281 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":633,"thread_start_us":517,"threads_started":1}
I20260928 16:50:28.085789 17132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:28.063043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52464 (local address 127.9.93.61:38393)
0928 16:50:28.075831 (+ 12788us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:28.075839 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:28.076845 (+ 1006us) server_negotiation.cc:407] Connection header received
0928 16:50:28.076929 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:28.076935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:28.077016 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:28.077115 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:50:28.078010 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:28.079861 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:28.082518 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:28.083864 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:28.083991 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:28.084120 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:28.085275 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0928 16:50:28.085424 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12627,"thread_start_us":12770,"threads_started":1}
I20260928 16:50:28.087110 17135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:28.072237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41929 (local address 127.0.0.1:51956)
0928 16:50:28.073598 (+ 1361us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:28.073621 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:28.073841 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:28.075605 (+ 1764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:28.075611 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:28.075667 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:50:28.075991 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:28.076005 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:28.077740 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:28.077745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:28.080021 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:28.080039 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:28.082183 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:28.082191 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:28.082406 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:28.082448 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 16:50:28.082529 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 16:50:28.082722 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1243,"thread_start_us":265,"threads_started":1}
I20260928 16:50:28.085156 17130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:28.062877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:38393 (local address 127.0.0.1:52464)
0928 16:50:28.076492 (+ 13615us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:28.076515 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:28.076704 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:28.077376 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:28.077382 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:28.077420 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:50:28.077777 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:28.077791 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:28.080115 (+ 2324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:28.080121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:28.082322 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:28.082338 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:28.084264 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:28.084270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:28.084473 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:28.084513 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 16:50:28.084621 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 16:50:28.084774 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13429,"thread_start_us":16681,"threads_started":1}
I20260928 16:50:28.088261 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:28.087764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.0.0.1:33240)
0928 16:50:28.087967 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:28.088072 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 16:50:28.088158 16407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cebda140f974905b3e3480b980f7a2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "31a246c9c6804fdb8430171f57199a93" is_pre_election: true
I20260928 16:50:28.088519 16407 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cebda140f974905b3e3480b980f7a2d in term 1.
I20260928 16:50:28.088455 16596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31a246c9c6804fdb8430171f57199a93" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6cebda140f974905b3e3480b980f7a2d" is_pre_election: true
W20260928 16:50:28.088605 16570 leader_election.cc:334] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2577aff545514d4389450abad8337d7b (127.9.93.62:33733): Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
I20260928 16:50:28.088847 16596 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a246c9c6804fdb8430171f57199a93 in term 1.
I20260928 16:50:28.089166 16572 leader_election.cc:302] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a246c9c6804fdb8430171f57199a93, 6cebda140f974905b3e3480b980f7a2d; no voters: 2577aff545514d4389450abad8337d7b
I20260928 16:50:28.089510 17128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 16:50:28.089502 16382 leader_election.cc:302] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a246c9c6804fdb8430171f57199a93, 6cebda140f974905b3e3480b980f7a2d; no voters:
I20260928 16:50:28.089637 17128 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2577aff545514d4389450abad8337d7b)
I20260928 16:50:28.089722 17128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:50:28.089918 17129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 16:50:28.090101 17129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2577aff545514d4389450abad8337d7b)
I20260928 16:50:28.090224 17129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:50:28.090334 17135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:28.089785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.0.0.1:33256)
0928 16:50:28.090029 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:28.090153 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260928 16:50:28.090565 16380 leader_election.cc:334] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2577aff545514d4389450abad8337d7b (127.9.93.62:33733): Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
I20260928 16:50:28.092371 17128 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: false } }
I20260928 16:50:28.092878 17129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: false } }
I20260928 16:50:28.093120 17128 leader_election.cc:288] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [CANDIDATE]: Term 2 election: Requested vote from peers 2577aff545514d4389450abad8337d7b (127.9.93.62:33733), 31a246c9c6804fdb8430171f57199a93 (127.9.93.61:38393)
I20260928 16:50:28.093887 17129 leader_election.cc:288] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [CANDIDATE]: Term 2 election: Requested vote from peers 2577aff545514d4389450abad8337d7b (127.9.93.62:33733), 6cebda140f974905b3e3480b980f7a2d (127.9.93.60:41929)
I20260928 16:50:28.093994 16407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cebda140f974905b3e3480b980f7a2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "31a246c9c6804fdb8430171f57199a93"
I20260928 16:50:28.094408 16407 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6cebda140f974905b3e3480b980f7a2d in current term 2: Already voted for candidate 31a246c9c6804fdb8430171f57199a93 in this term.
I20260928 16:50:28.094383 17130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:28.093301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.0.0.1:33272)
0928 16:50:28.093530 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:28.093646 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 16:50:28.094578 17135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:28.093915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.0.0.1:33274)
0928 16:50:28.094180 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:28.094279 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260928 16:50:28.094383 16596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31a246c9c6804fdb8430171f57199a93" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6cebda140f974905b3e3480b980f7a2d"
I20260928 16:50:28.094924 16596 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 31a246c9c6804fdb8430171f57199a93 in current term 2: Already voted for candidate 6cebda140f974905b3e3480b980f7a2d in this term.
I20260928 16:50:28.095844 17135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:28.095292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.0.0.1:33284)
0928 16:50:28.095597 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:28.095696 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260928 16:50:28.095845 17130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:28.095422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.0.0.1:33298)
0928 16:50:28.095597 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:28.095707 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 16:50:28.096048 16570 leader_election.cc:334] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2577aff545514d4389450abad8337d7b (127.9.93.62:33733): Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
W20260928 16:50:28.096066 16380 leader_election.cc:334] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2577aff545514d4389450abad8337d7b (127.9.93.62:33733): Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
I20260928 16:50:28.096189 16380 leader_election.cc:302] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a246c9c6804fdb8430171f57199a93; no voters: 2577aff545514d4389450abad8337d7b, 6cebda140f974905b3e3480b980f7a2d
I20260928 16:50:28.096189 16570 leader_election.cc:302] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cebda140f974905b3e3480b980f7a2d; no voters: 2577aff545514d4389450abad8337d7b, 31a246c9c6804fdb8430171f57199a93
I20260928 16:50:28.096501 17128 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 16:50:28.096684 17129 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 16:50:28.544123 17138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:28.543243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.1:43427)
0928 16:50:28.543833 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:28.543944 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":346,"threads_started":1}
I20260928 16:50:28.550002 17139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:28.549037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.2:51705)
0928 16:50:28.549623 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:28.549747 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":321,"threads_started":1}
I20260928 16:50:28.551014 17140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:28.550145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.3:40943)
0928 16:50:28.550689 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:28.550788 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":373,"threads_started":1}
I20260928 16:50:29.546043 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:29.545121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.1:42677)
0928 16:50:29.545698 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:29.545802 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":254,"threads_started":1}
I20260928 16:50:29.552362 17142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:29.551161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.2:45113)
0928 16:50:29.551989 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:29.552130 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":916,"threads_started":1}
I20260928 16:50:29.553337 17143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:29.551915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.3:41793)
0928 16:50:29.553001 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:29.553119 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":967,"thread_start_us":845,"threads_started":1}
I20260928 16:50:29.786327 17144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:50:29.786677 17144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: false } }
I20260928 16:50:29.787648 17144 leader_election.cc:288] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2577aff545514d4389450abad8337d7b (127.9.93.62:33733), 31a246c9c6804fdb8430171f57199a93 (127.9.93.61:38393)
I20260928 16:50:29.788439 16407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cebda140f974905b3e3480b980f7a2d" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "31a246c9c6804fdb8430171f57199a93" is_pre_election: true
I20260928 16:50:29.788738 16407 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cebda140f974905b3e3480b980f7a2d in term 2.
I20260928 16:50:29.788973 17145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:29.787922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.0.0.1:33542)
0928 16:50:29.788618 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:29.788752 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":428,"threads_started":1}
I20260928 16:50:29.789304 16572 leader_election.cc:302] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a246c9c6804fdb8430171f57199a93, 6cebda140f974905b3e3480b980f7a2d; no voters:
I20260928 16:50:29.789605 17144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260928 16:50:29.789724 17144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:50:29.789826 17144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 2 FOLLOWER]: Advancing to term 3
I20260928 16:50:29.790585 17145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:29.790189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.0.0.1:33548)
0928 16:50:29.790356 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:29.790436 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 16:50:29.790796 16570 leader_election.cc:334] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2577aff545514d4389450abad8337d7b (127.9.93.62:33733): Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
I20260928 16:50:29.792411 17144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: false } }
I20260928 16:50:29.793030 17144 leader_election.cc:288] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [CANDIDATE]: Term 3 election: Requested vote from peers 2577aff545514d4389450abad8337d7b (127.9.93.62:33733), 31a246c9c6804fdb8430171f57199a93 (127.9.93.61:38393)
I20260928 16:50:29.793627 17145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:29.793184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.0.0.1:33562)
0928 16:50:29.793345 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:29.793442 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 16:50:29.793696 16407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cebda140f974905b3e3480b980f7a2d" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "31a246c9c6804fdb8430171f57199a93"
I20260928 16:50:29.793954 16407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 2 FOLLOWER]: Advancing to term 3
I20260928 16:50:29.794888 17145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:29.794495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.0.0.1:33572)
0928 16:50:29.794658 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:29.794744 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 16:50:29.795081 16570 leader_election.cc:334] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 2577aff545514d4389450abad8337d7b (127.9.93.62:33733): Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
I20260928 16:50:29.796650 16407 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cebda140f974905b3e3480b980f7a2d in term 3.
I20260928 16:50:29.797106 16572 leader_election.cc:302] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a246c9c6804fdb8430171f57199a93, 6cebda140f974905b3e3480b980f7a2d; no voters: 2577aff545514d4389450abad8337d7b
I20260928 16:50:29.797405 17144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 3 FOLLOWER]: Leader election won for term 3
I20260928 16:50:29.797873 17144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [term 3 LEADER]: Becoming Leader. State: Replica: 6cebda140f974905b3e3480b980f7a2d, State: Running, Role: LEADER
I20260928 16:50:29.798308 17144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: false } }
I20260928 16:50:29.800290 17147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cebda140f974905b3e3480b980f7a2d. Latest consensus state: current_term: 3 leader_uuid: "6cebda140f974905b3e3480b980f7a2d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: false } } }
I20260928 16:50:29.800562 17147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:29.801218 17149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:50:29.805891 17149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:50:29.806866 17149 catalog_manager.cc:1259] Loaded cluster ID: 5bcd09d1c1b14d24a6129be96c99a994
I20260928 16:50:29.806957 17149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:50:29.808367 17149 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:50:29.809226 17149 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d: Loaded TSK: 0
I20260928 16:50:29.810199 17149 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:50:29.854736 16585 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:50:29.862860 16585 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:50:29.868317 17145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:29.867732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.0.0.1:33602)
0928 16:50:29.867979 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:29.868101 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260928 16:50:29.868415 16407 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [term 3 FOLLOWER]: Refusing update from remote peer 6cebda140f974905b3e3480b980f7a2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260928 16:50:29.869350 17144 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 16:50:29.869630 17145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:29.869150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.0.0.1:33606)
0928 16:50:29.869348 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:29.869452 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 16:50:29.869891 16570 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d -> Peer 2577aff545514d4389450abad8337d7b (127.9.93.62:33733): Couldn't send request to peer 2577aff545514d4389450abad8337d7b. Status: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 16:50:29.873060 17151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cebda140f974905b3e3480b980f7a2d. Latest consensus state: current_term: 3 leader_uuid: "6cebda140f974905b3e3480b980f7a2d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: false } } }
I20260928 16:50:29.873395 17151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:50:29.874938 17144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "6cebda140f974905b3e3480b980f7a2d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: false } } }
I20260928 16:50:29.875345 17144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:29.875892 17151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "6cebda140f974905b3e3480b980f7a2d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2577aff545514d4389450abad8337d7b" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33733 } } peers { permanent_uuid: "31a246c9c6804fdb8430171f57199a93" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "6cebda140f974905b3e3480b980f7a2d" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41929 } attrs { promote: false } } }
I20260928 16:50:29.876197 17151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a246c9c6804fdb8430171f57199a93 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:50:29.903410 17145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:29.885259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:43275 (local address 127.0.0.1:54754)
0928 16:50:29.885424 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:29.885449 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:29.885582 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:29.893730 (+ 8148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:29.893738 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:29.893785 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:50:29.894276 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:29.894294 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:29.898148 (+ 3854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:29.898155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:29.900668 (+ 2513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:29.900689 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:29.902508 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:29.902515 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:29.902710 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:29.902754 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 16:50:29.902828 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 16:50:29.902993 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":29}
I20260928 16:50:29.904763 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:29.885388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54754 (local address 127.9.93.2:43275)
0928 16:50:29.892331 (+ 6943us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:29.892341 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:29.892382 (+ 41us) server_negotiation.cc:407] Connection header received
0928 16:50:29.892508 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:29.892515 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:29.892630 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:29.894535 (+ 1905us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:50:29.894815 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:29.896104 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:29.900901 (+ 4797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:29.902283 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:29.904015 (+ 1732us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:29.904160 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:29.904228 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 16:50:29.904397 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6652,"thread_start_us":273,"threads_started":1}
I20260928 16:50:29.905613 16847 tablet_service.cc:1467] Processing CreateTablet for tablet 9e7d297179524de9a39fce3dee5dd2e5 (DEFAULT_TABLE table=test-table [id=f535d445c6784f6cbfa84849b6848a58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:50:29.906108 16847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7d297179524de9a39fce3dee5dd2e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:29.908061 17161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:29.886143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37806 (local address 127.9.93.3:42893)
0928 16:50:29.897097 (+ 10954us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:29.897106 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:29.898691 (+ 1585us) server_negotiation.cc:407] Connection header received
0928 16:50:29.898792 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:29.898797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:29.898889 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:29.899012 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:50:29.899823 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:29.900594 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:29.902971 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:29.904254 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:29.904354 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:29.904458 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:29.907494 (+ 3036us) server_negotiation.cc:299] Negotiation successful
0928 16:50:29.907732 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10764,"thread_start_us":11254,"threads_started":1}
I20260928 16:50:29.908644 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:29.885910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:42893 (local address 127.0.0.1:37806)
0928 16:50:29.898386 (+ 12476us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:29.898410 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:29.898564 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:29.899061 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:29.899067 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:29.899098 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:50:29.899616 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:29.899629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:29.900751 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:29.900755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:29.902776 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:29.902788 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:29.904631 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:29.904637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:29.904828 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:29.904863 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:50:29.904954 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 16:50:29.905097 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12324,"mutex_wait_us":3,"thread_start_us":10878,"threads_started":1}
I20260928 16:50:29.909674 17159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:29.885518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:40577 (local address 127.0.0.1:52616)
0928 16:50:29.897134 (+ 11616us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:29.897157 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:29.897312 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:29.903398 (+ 6086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:29.903404 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:29.903446 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:50:29.903807 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:29.903819 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:29.905215 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:29.905222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:29.906656 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:29.906669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:29.908716 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:29.908723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:29.909093 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:29.909129 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 16:50:29.909216 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 16:50:29.909370 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11442,"thread_start_us":12551,"threads_started":1}
I20260928 16:50:29.909762 16988 tablet_service.cc:1467] Processing CreateTablet for tablet 9e7d297179524de9a39fce3dee5dd2e5 (DEFAULT_TABLE table=test-table [id=f535d445c6784f6cbfa84849b6848a58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:50:29.910312 16988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7d297179524de9a39fce3dee5dd2e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:29.910404 17160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:29.885972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52616 (local address 127.9.93.1:40577)
0928 16:50:29.902933 (+ 16961us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:29.902940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:29.902962 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:50:29.903043 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:29.903048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:29.903136 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:29.903251 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:50:29.904254 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:29.905028 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:29.906832 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:29.908450 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:29.908604 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:29.908724 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:29.910009 (+ 1285us) server_negotiation.cc:299] Negotiation successful
0928 16:50:29.910152 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16804,"thread_start_us":17957,"threads_started":1}
I20260928 16:50:29.911190 16700 tablet_service.cc:1467] Processing CreateTablet for tablet 9e7d297179524de9a39fce3dee5dd2e5 (DEFAULT_TABLE table=test-table [id=f535d445c6784f6cbfa84849b6848a58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:50:29.911672 16700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7d297179524de9a39fce3dee5dd2e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:29.917515 17163 tablet_bootstrap.cc:492] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b: Bootstrap starting.
I20260928 16:50:29.919703 17163 tablet_bootstrap.cc:654] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:29.920817 17162 tablet_bootstrap.cc:492] T 9e7d297179524de9a39fce3dee5dd2e5 P d29db76b68474ab6aa678633dd238fe1: Bootstrap starting.
I20260928 16:50:29.922361 17163 tablet_bootstrap.cc:492] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b: No bootstrap required, opened a new log
I20260928 16:50:29.922538 17163 ts_tablet_manager.cc:1397] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 16:50:29.923223 17162 tablet_bootstrap.cc:654] T 9e7d297179524de9a39fce3dee5dd2e5 P d29db76b68474ab6aa678633dd238fe1: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:29.923280 17163 raft_consensus.cc:348] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7397325a370249b5a66a0177c1e2fa13" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42893 } } peers { permanent_uuid: "d29db76b68474ab6aa678633dd238fe1" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43275 } } peers { permanent_uuid: "8c5622e16bb5462793db42b715d8d31b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40577 } }
I20260928 16:50:29.923755 17163 raft_consensus.cc:374] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:29.923859 17163 raft_consensus.cc:729] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c5622e16bb5462793db42b715d8d31b, State: Initialized, Role: FOLLOWER
I20260928 16:50:29.924139 17163 consensus_queue.cc:260] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7397325a370249b5a66a0177c1e2fa13" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42893 } } peers { permanent_uuid: "d29db76b68474ab6aa678633dd238fe1" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43275 } } peers { permanent_uuid: "8c5622e16bb5462793db42b715d8d31b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40577 } }
I20260928 16:50:29.925071 17163 ts_tablet_manager.cc:1428] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 16:50:29.926286 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:29.925764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.1:49549)
0928 16:50:29.926003 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:29.926110 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260928 16:50:29.929924 16765 heartbeater.cc:503] Master 127.9.93.60:41929 was elected leader, sending a full tablet report...
I20260928 16:50:29.932022 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:29.931439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.1:38487)
0928 16:50:29.931728 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:29.931829 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260928 16:50:29.932230 17164 tablet_bootstrap.cc:492] T 9e7d297179524de9a39fce3dee5dd2e5 P 7397325a370249b5a66a0177c1e2fa13: Bootstrap starting.
I20260928 16:50:29.933755 17162 tablet_bootstrap.cc:492] T 9e7d297179524de9a39fce3dee5dd2e5 P d29db76b68474ab6aa678633dd238fe1: No bootstrap required, opened a new log
I20260928 16:50:29.933924 17162 ts_tablet_manager.cc:1397] T 9e7d297179524de9a39fce3dee5dd2e5 P d29db76b68474ab6aa678633dd238fe1: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260928 16:50:29.934005 17164 tablet_bootstrap.cc:654] T 9e7d297179524de9a39fce3dee5dd2e5 P 7397325a370249b5a66a0177c1e2fa13: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:29.934746 17162 raft_consensus.cc:348] T 9e7d297179524de9a39fce3dee5dd2e5 P d29db76b68474ab6aa678633dd238fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7397325a370249b5a66a0177c1e2fa13" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42893 } } peers { permanent_uuid: "d29db76b68474ab6aa678633dd238fe1" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43275 } } peers { permanent_uuid: "8c5622e16bb5462793db42b715d8d31b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40577 } }
I20260928 16:50:29.935065 17162 raft_consensus.cc:374] T 9e7d297179524de9a39fce3dee5dd2e5 P d29db76b68474ab6aa678633dd238fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:29.935173 17162 raft_consensus.cc:729] T 9e7d297179524de9a39fce3dee5dd2e5 P d29db76b68474ab6aa678633dd238fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d29db76b68474ab6aa678633dd238fe1, State: Initialized, Role: FOLLOWER
I20260928 16:50:29.935503 17162 consensus_queue.cc:260] T 9e7d297179524de9a39fce3dee5dd2e5 P d29db76b68474ab6aa678633dd238fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7397325a370249b5a66a0177c1e2fa13" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42893 } } peers { permanent_uuid: "d29db76b68474ab6aa678633dd238fe1" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43275 } } peers { permanent_uuid: "8c5622e16bb5462793db42b715d8d31b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40577 } }
I20260928 16:50:29.936349 17164 tablet_bootstrap.cc:492] T 9e7d297179524de9a39fce3dee5dd2e5 P 7397325a370249b5a66a0177c1e2fa13: No bootstrap required, opened a new log
I20260928 16:50:29.936480 17162 ts_tablet_manager.cc:1428] T 9e7d297179524de9a39fce3dee5dd2e5 P d29db76b68474ab6aa678633dd238fe1: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 16:50:29.936607 17164 ts_tablet_manager.cc:1397] T 9e7d297179524de9a39fce3dee5dd2e5 P 7397325a370249b5a66a0177c1e2fa13: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260928 16:50:29.937281 17142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:29.936824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.2:40069)
0928 16:50:29.937002 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:29.937112 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 16:50:29.937922 17164 raft_consensus.cc:348] T 9e7d297179524de9a39fce3dee5dd2e5 P 7397325a370249b5a66a0177c1e2fa13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7397325a370249b5a66a0177c1e2fa13" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42893 } } peers { permanent_uuid: "d29db76b68474ab6aa678633dd238fe1" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43275 } } peers { permanent_uuid: "8c5622e16bb5462793db42b715d8d31b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40577 } }
I20260928 16:50:29.938261 17164 raft_consensus.cc:374] T 9e7d297179524de9a39fce3dee5dd2e5 P 7397325a370249b5a66a0177c1e2fa13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:29.938390 17164 raft_consensus.cc:729] T 9e7d297179524de9a39fce3dee5dd2e5 P 7397325a370249b5a66a0177c1e2fa13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7397325a370249b5a66a0177c1e2fa13, State: Initialized, Role: FOLLOWER
I20260928 16:50:29.938787 17164 consensus_queue.cc:260] T 9e7d297179524de9a39fce3dee5dd2e5 P 7397325a370249b5a66a0177c1e2fa13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7397325a370249b5a66a0177c1e2fa13" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42893 } } peers { permanent_uuid: "d29db76b68474ab6aa678633dd238fe1" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43275 } } peers { permanent_uuid: "8c5622e16bb5462793db42b715d8d31b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40577 } }
I20260928 16:50:29.939795 16912 heartbeater.cc:503] Master 127.9.93.60:41929 was elected leader, sending a full tablet report...
I20260928 16:50:29.941586 17142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:29.941123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.2:57043)
0928 16:50:29.941288 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:29.941379 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 16:50:29.941991 17143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:29.940722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.3:43585)
0928 16:50:29.940894 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:29.941003 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 16:50:29.940402 17164 ts_tablet_manager.cc:1428] T 9e7d297179524de9a39fce3dee5dd2e5 P 7397325a370249b5a66a0177c1e2fa13: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260928 16:50:29.942384 17053 heartbeater.cc:503] Master 127.9.93.60:41929 was elected leader, sending a full tablet report...
I20260928 16:50:29.943676 17143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:29.942720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.3:39421)
0928 16:50:29.943412 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:29.943501 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523}
I20260928 16:50:30.017872 17166 raft_consensus.cc:482] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:50:30.018200 17166 raft_consensus.cc:504] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7397325a370249b5a66a0177c1e2fa13" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42893 } } peers { permanent_uuid: "d29db76b68474ab6aa678633dd238fe1" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43275 } } peers { permanent_uuid: "8c5622e16bb5462793db42b715d8d31b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40577 } }
I20260928 16:50:30.018965 17166 leader_election.cc:288] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7397325a370249b5a66a0177c1e2fa13 (127.9.93.3:42893), d29db76b68474ab6aa678633dd238fe1 (127.9.93.2:43275)
I20260928 16:50:30.019850 17008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7d297179524de9a39fce3dee5dd2e5" candidate_uuid: "8c5622e16bb5462793db42b715d8d31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7397325a370249b5a66a0177c1e2fa13" is_pre_election: true
I20260928 16:50:30.019956 16867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7d297179524de9a39fce3dee5dd2e5" candidate_uuid: "8c5622e16bb5462793db42b715d8d31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d29db76b68474ab6aa678633dd238fe1" is_pre_election: true
I20260928 16:50:30.020153 17008 raft_consensus.cc:2454] T 9e7d297179524de9a39fce3dee5dd2e5 P 7397325a370249b5a66a0177c1e2fa13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c5622e16bb5462793db42b715d8d31b in term 0.
I20260928 16:50:30.020289 16867 raft_consensus.cc:2454] T 9e7d297179524de9a39fce3dee5dd2e5 P d29db76b68474ab6aa678633dd238fe1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c5622e16bb5462793db42b715d8d31b in term 0.
I20260928 16:50:30.020787 16655 leader_election.cc:302] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7397325a370249b5a66a0177c1e2fa13, 8c5622e16bb5462793db42b715d8d31b; no voters:
I20260928 16:50:30.021185 17166 raft_consensus.cc:2781] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:50:30.021327 17166 raft_consensus.cc:482] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:50:30.021428 17166 raft_consensus.cc:3037] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:30.024047 17166 raft_consensus.cc:504] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7397325a370249b5a66a0177c1e2fa13" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42893 } } peers { permanent_uuid: "d29db76b68474ab6aa678633dd238fe1" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43275 } } peers { permanent_uuid: "8c5622e16bb5462793db42b715d8d31b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40577 } }
I20260928 16:50:30.024777 17166 leader_election.cc:288] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b [CANDIDATE]: Term 1 election: Requested vote from peers 7397325a370249b5a66a0177c1e2fa13 (127.9.93.3:42893), d29db76b68474ab6aa678633dd238fe1 (127.9.93.2:43275)
I20260928 16:50:30.025462 17008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7d297179524de9a39fce3dee5dd2e5" candidate_uuid: "8c5622e16bb5462793db42b715d8d31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7397325a370249b5a66a0177c1e2fa13"
I20260928 16:50:30.025540 16867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7d297179524de9a39fce3dee5dd2e5" candidate_uuid: "8c5622e16bb5462793db42b715d8d31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d29db76b68474ab6aa678633dd238fe1"
I20260928 16:50:30.025758 17008 raft_consensus.cc:3037] T 9e7d297179524de9a39fce3dee5dd2e5 P 7397325a370249b5a66a0177c1e2fa13 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:30.025789 16867 raft_consensus.cc:3037] T 9e7d297179524de9a39fce3dee5dd2e5 P d29db76b68474ab6aa678633dd238fe1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:30.029201 16867 raft_consensus.cc:2454] T 9e7d297179524de9a39fce3dee5dd2e5 P d29db76b68474ab6aa678633dd238fe1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c5622e16bb5462793db42b715d8d31b in term 1.
I20260928 16:50:30.029196 17008 raft_consensus.cc:2454] T 9e7d297179524de9a39fce3dee5dd2e5 P 7397325a370249b5a66a0177c1e2fa13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c5622e16bb5462793db42b715d8d31b in term 1.
I20260928 16:50:30.030040 16655 leader_election.cc:302] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7397325a370249b5a66a0177c1e2fa13, 8c5622e16bb5462793db42b715d8d31b, d29db76b68474ab6aa678633dd238fe1; no voters:
I20260928 16:50:30.030524 17166 raft_consensus.cc:2781] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:50:30.031008 17166 raft_consensus.cc:686] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b [term 1 LEADER]: Becoming Leader. State: Replica: 8c5622e16bb5462793db42b715d8d31b, State: Running, Role: LEADER
I20260928 16:50:30.031432 17166 consensus_queue.cc:237] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7397325a370249b5a66a0177c1e2fa13" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42893 } } peers { permanent_uuid: "d29db76b68474ab6aa678633dd238fe1" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43275 } } peers { permanent_uuid: "8c5622e16bb5462793db42b715d8d31b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40577 } }
I20260928 16:50:30.032271 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:30.031766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.1:32841)
0928 16:50:30.031960 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:30.032083 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 16:50:30.034992 16585 catalog_manager.cc:5495] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c5622e16bb5462793db42b715d8d31b (127.9.93.1). New cstate: current_term: 1 leader_uuid: "8c5622e16bb5462793db42b715d8d31b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7397325a370249b5a66a0177c1e2fa13" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d29db76b68474ab6aa678633dd238fe1" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c5622e16bb5462793db42b715d8d31b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40577 } health_report { overall_health: HEALTHY } } }
W20260928 16:50:30.065515 16773 tablet.cc:2369] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:50:30.069898 17160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:30.064152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52620 (local address 127.9.93.1:40577)
0928 16:50:30.064434 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:30.064441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:30.064465 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:50:30.064543 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:30.064548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:30.064659 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:30.064747 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:50:30.065165 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:30.066175 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:30.067636 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:30.068179 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:30.068319 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:30.068816 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:50:30.068942 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:50:30.069017 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:30.069246 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 16:50:30.069338 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 16:50:30.073954 16867 raft_consensus.cc:1261] T 9e7d297179524de9a39fce3dee5dd2e5 P d29db76b68474ab6aa678633dd238fe1 [term 1 FOLLOWER]: Refusing update from remote peer 8c5622e16bb5462793db42b715d8d31b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:50:30.073954 17008 raft_consensus.cc:1261] T 9e7d297179524de9a39fce3dee5dd2e5 P 7397325a370249b5a66a0177c1e2fa13 [term 1 FOLLOWER]: Refusing update from remote peer 8c5622e16bb5462793db42b715d8d31b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:50:30.074653 17166 consensus_queue.cc:1035] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7397325a370249b5a66a0177c1e2fa13" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:50:30.075084 17171 consensus_queue.cc:1035] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d29db76b68474ab6aa678633dd238fe1" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 43275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:50:30.077589 17143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:30.076901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.3:59435)
0928 16:50:30.077246 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:30.077370 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210}
I20260928 16:50:30.078902 17142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:30.078282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.2:50487)
0928 16:50:30.078576 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:30.078689 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260928 16:50:30.084952 17143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:30.083863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.3:56501)
0928 16:50:30.084585 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:30.084709 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":566}
I20260928 16:50:30.087468 17176 mvcc.cc:205] Tried to move back new op lower bound from 7334355886375391232 to 7334355886211903488. Current Snapshot: MvccSnapshot[applied={T|T < 7334355886375391232}]
I20260928 16:50:30.089759 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:30.089154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.1:49735)
0928 16:50:30.089405 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:30.089529 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260928 16:50:30.090149 17142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:30.088143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.9.93.2:56219)
0928 16:50:30.089774 (+ 1631us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:30.089898 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1488}
I20260928 16:50:30.093972 17175 mvcc.cc:205] Tried to move back new op lower bound from 7334355886375391232 to 7334355886211903488. Current Snapshot: MvccSnapshot[applied={T|T < 7334355886375391232}]
I20260928 16:50:30.095825 16585 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:50:30.096194 16585 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:50:30.104368 16585 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6cebda140f974905b3e3480b980f7a2d: Sending DeleteTablet for 3 replicas of tablet 9e7d297179524de9a39fce3dee5dd2e5
I20260928 16:50:30.105903 16700 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e7d297179524de9a39fce3dee5dd2e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:30 UTC) from {username='oryx', principal='oryx/127.9.93.60@KRBTEST.COM'} at 127.0.0.1:52616
I20260928 16:50:30.105937 16846 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e7d297179524de9a39fce3dee5dd2e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:30 UTC) from {username='oryx', principal='oryx/127.9.93.60@KRBTEST.COM'} at 127.0.0.1:54754
I20260928 16:50:30.106294 16847 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e7d297179524de9a39fce3dee5dd2e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:30 UTC) from {username='oryx', principal='oryx/127.9.93.60@KRBTEST.COM'} at 127.0.0.1:54754
I20260928 16:50:30.106565 16987 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e7d297179524de9a39fce3dee5dd2e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:30 UTC) from {username='oryx', principal='oryx/127.9.93.60@KRBTEST.COM'} at 127.0.0.1:37806
I20260928 16:50:30.106583 16988 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e7d297179524de9a39fce3dee5dd2e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:30 UTC) from {username='oryx', principal='oryx/127.9.93.60@KRBTEST.COM'} at 127.0.0.1:37806
I20260928 16:50:30.107336 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 16558
I20260928 16:50:30.111204 17184 tablet_replica.cc:330] T 9e7d297179524de9a39fce3dee5dd2e5 P 7397325a370249b5a66a0177c1e2fa13: stopping tablet replica
I20260928 16:50:30.111845 17184 raft_consensus.cc:2229] T 9e7d297179524de9a39fce3dee5dd2e5 P 7397325a370249b5a66a0177c1e2fa13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:50:30.112306 17184 raft_consensus.cc:2258] T 9e7d297179524de9a39fce3dee5dd2e5 P 7397325a370249b5a66a0177c1e2fa13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:30.113754 17184 ts_tablet_manager.cc:1905] T 9e7d297179524de9a39fce3dee5dd2e5 P 7397325a370249b5a66a0177c1e2fa13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:50:30.120200 17182 tablet_replica.cc:330] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b: stopping tablet replica
I20260928 16:50:30.120891 17182 raft_consensus.cc:2229] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:50:30.121857 17182 raft_consensus.cc:2258] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:30.122716 17184 ts_tablet_manager.cc:1918] T 9e7d297179524de9a39fce3dee5dd2e5 P 7397325a370249b5a66a0177c1e2fa13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:50:30.123551 17184 log.cc:1196] T 9e7d297179524de9a39fce3dee5dd2e5 P 7397325a370249b5a66a0177c1e2fa13: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/9e7d297179524de9a39fce3dee5dd2e5
I20260928 16:50:30.124130 17184 ts_tablet_manager.cc:1939] T 9e7d297179524de9a39fce3dee5dd2e5 P 7397325a370249b5a66a0177c1e2fa13: Deleting consensus metadata
I20260928 16:50:30.124351 17182 ts_tablet_manager.cc:1905] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:50:30.125883 16572 catalog_manager.cc:4841] TS 7397325a370249b5a66a0177c1e2fa13 (127.9.93.3:42893): tablet 9e7d297179524de9a39fce3dee5dd2e5 (table test-table [id=f535d445c6784f6cbfa84849b6848a58]) successfully deleted
W20260928 16:50:30.126494 16572 catalog_manager.cc:4808] TS 7397325a370249b5a66a0177c1e2fa13 (127.9.93.3:42893): delete failed for tablet 9e7d297179524de9a39fce3dee5dd2e5 because the tablet was not found. No further retry: Not found: Tablet not found: 9e7d297179524de9a39fce3dee5dd2e5
I20260928 16:50:30.128034 17183 tablet_replica.cc:330] T 9e7d297179524de9a39fce3dee5dd2e5 P d29db76b68474ab6aa678633dd238fe1: stopping tablet replica
I20260928 16:50:30.128304 17183 raft_consensus.cc:2229] T 9e7d297179524de9a39fce3dee5dd2e5 P d29db76b68474ab6aa678633dd238fe1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:50:30.128682 17183 raft_consensus.cc:2258] T 9e7d297179524de9a39fce3dee5dd2e5 P d29db76b68474ab6aa678633dd238fe1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:30.129905 17183 ts_tablet_manager.cc:1905] T 9e7d297179524de9a39fce3dee5dd2e5 P d29db76b68474ab6aa678633dd238fe1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:50:30.131729 17182 ts_tablet_manager.cc:1918] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:50:30.131970 17182 log.cc:1196] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-0/wal/wals/9e7d297179524de9a39fce3dee5dd2e5
I20260928 16:50:30.132575 17182 ts_tablet_manager.cc:1939] T 9e7d297179524de9a39fce3dee5dd2e5 P 8c5622e16bb5462793db42b715d8d31b: Deleting consensus metadata
I20260928 16:50:30.133994 16573 catalog_manager.cc:4841] TS 8c5622e16bb5462793db42b715d8d31b (127.9.93.1:40577): tablet 9e7d297179524de9a39fce3dee5dd2e5 (table test-table [id=f535d445c6784f6cbfa84849b6848a58]) successfully deleted
I20260928 16:50:30.136521 17183 ts_tablet_manager.cc:1918] T 9e7d297179524de9a39fce3dee5dd2e5 P d29db76b68474ab6aa678633dd238fe1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:50:30.136688 17183 log.cc:1196] T 9e7d297179524de9a39fce3dee5dd2e5 P d29db76b68474ab6aa678633dd238fe1: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/9e7d297179524de9a39fce3dee5dd2e5
I20260928 16:50:30.137149 17183 ts_tablet_manager.cc:1939] T 9e7d297179524de9a39fce3dee5dd2e5 P d29db76b68474ab6aa678633dd238fe1: Deleting consensus metadata
I20260928 16:50:30.138722 16571 catalog_manager.cc:4841] TS d29db76b68474ab6aa678633dd238fe1 (127.9.93.2:43275): tablet 9e7d297179524de9a39fce3dee5dd2e5 (table test-table [id=f535d445c6784f6cbfa84849b6848a58]) successfully deleted
W20260928 16:50:30.139164 16571 catalog_manager.cc:4808] TS d29db76b68474ab6aa678633dd238fe1 (127.9.93.2:43275): delete failed for tablet 9e7d297179524de9a39fce3dee5dd2e5 because the tablet was not found. No further retry: Not found: Tablet not found: 9e7d297179524de9a39fce3dee5dd2e5
I20260928 16:50:30.143855 17160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:30.110336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52624 (local address 127.9.93.1:40577)
0928 16:50:30.110563 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:30.110571 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:30.123547 (+ 12976us) server_negotiation.cc:407] Connection header received
0928 16:50:30.127552 (+ 4005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:30.127561 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:30.127699 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:30.127815 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:30.129720 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:30.130696 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:30.132169 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:30.132681 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:30.132837 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:30.136007 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:30.136031 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:30.136055 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:30.136109 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:30.140050 (+ 3941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:30.141035 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:30.141045 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:30.141048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:30.141236 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:30.141749 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:30.141754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:30.141756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:30.142337 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:30.142629 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:30.143163 (+ 534us) server_negotiation.cc:299] Negotiation successful
0928 16:50:30.143474 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:50:30.144690 16760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:30.291366 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 16558
I20260928 16:50:30.309897 17159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:30.309381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.0.0.1:33622)
0928 16:50:30.309621 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:30.309734 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 16:50:30.311065 17159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:30.310660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.0.0.1:33628)
0928 16:50:30.310822 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:30.310912 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 16:50:30.313647 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 16777
I20260928 16:50:30.330111 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:30.314436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54760 (local address 127.9.93.2:43275)
0928 16:50:30.314664 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:30.314671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:30.314695 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:50:30.314876 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:30.314881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:30.314977 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:30.315088 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:30.316612 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:30.317396 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:30.318774 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:30.319245 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:30.319443 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:30.322322 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:30.322356 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:30.322381 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:30.322442 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:30.326490 (+ 4048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:30.327537 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:30.327558 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:30.327562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:30.327785 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:30.328301 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:30.328305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:30.328308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:30.328880 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:30.329242 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:30.329538 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 16:50:30.329770 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 16:50:30.330853 16907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:30.435748 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 16777
I20260928 16:50:30.452935 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 16925
I20260928 16:50:30.468308 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:30.453853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37810 (local address 127.9.93.3:42893)
0928 16:50:30.454390 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:30.454396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:30.454414 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:50:30.454484 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:30.454488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:30.454570 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:30.454679 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:30.456263 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:30.457087 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:30.458530 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:30.459040 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:30.459169 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:30.461982 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:30.462005 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:30.462031 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:30.462094 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:30.465207 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:30.465993 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:30.466003 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:30.466005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:30.466197 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:30.466613 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:30.466618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:30.466620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:30.467081 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:30.467467 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:30.467777 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 16:50:30.468003 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":278,"threads_started":1}
I20260928 16:50:30.469120 17048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:30.571100 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 16925
I20260928 16:50:30.586982 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 16256
I20260928 16:50:30.588431 16427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:30.722611 17159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:30.721896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.0.0.1:33644)
0928 16:50:30.722220 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:30.722339 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260928 16:50:30.723413 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 16256
I20260928 16:50:30.724067 17159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:30.723546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33733 (local address 127.0.0.1:33660)
0928 16:50:30.723789 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:30.723888 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260928 16:50:30.743507 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 16353
I20260928 16:50:30.744763 16616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:30.882788 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 16353
2026-09-28T16:50:30Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11980 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:50:30 dist-test-slave-wzcb krb5kdc[17194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:50:30 dist-test-slave-wzcb krb5kdc[17194](info): set up 2 sockets
Sep 28 16:50:30 dist-test-slave-wzcb krb5kdc[17194](info): commencing operation
krb5kdc: starting...
W20260928 16:50:31.922617 9588 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.015s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:50:32 dist-test-slave-wzcb krb5kdc[17194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:50:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:50:32Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.62.keytab.
Entry for principal oryx/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.62.keytab.
I20260928 16:50:32.179409 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:33613
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.62.keytab
--principal=oryx/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:42491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:33613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:50:32.526496 17207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:32.526857 17207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:32.527004 17207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:32.534541 17207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:50:32.534687 17207 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 16:50:32.534762 17207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:32.534859 17207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:32.534927 17207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:50:32.535017 17207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:50:32.545522 17207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:42491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:33613
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.62.keytab
--principal=oryx/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:33613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:32.546983 17207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:32.548539 17207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:32.561354 17215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:32.561390 17213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:32.561710 17207 server_base.cc:1034] running on GCE node
W20260928 16:50:32.561453 17212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:32.562918 17207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:32.564641 17207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:32.565919 17207 hybrid_clock.cc:648] HybridClock initialized: now 1790614232565838 us; error 49 us; skew 500 ppm
I20260928 16:50:32.566288 17207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:32 dist-test-slave-wzcb krb5kdc[17194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614232, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:32.569768 17207 init.cc:377] Logged in from keytab as oryx/127.9.93.62@KRBTEST.COM (short username oryx)
I20260928 16:50:32.571446 17207 webserver.cc:460] Webserver started at http://127.9.93.62:39623/ using document root <none> and password file <none>
I20260928 16:50:32.572302 17207 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:32.572402 17207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:32.572759 17207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:32.576042 17207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "58662c670a204a0d9fbf0403a4ea1822"
format_stamp: "Formatted at 2026-09-28 16:50:32 on dist-test-slave-wzcb"
server_key: "c3daa58289348f35e423e487feca1c9d"
server_key_iv: "2a8faf34d0730673507f3a8723ff658a"
server_key_version: "encryptionkey@0"
I20260928 16:50:32.576910 17207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "58662c670a204a0d9fbf0403a4ea1822"
format_stamp: "Formatted at 2026-09-28 16:50:32 on dist-test-slave-wzcb"
server_key: "c3daa58289348f35e423e487feca1c9d"
server_key_iv: "2a8faf34d0730673507f3a8723ff658a"
server_key_version: "encryptionkey@0"
I20260928 16:50:32.582481 17207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:50:32.586364 17222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:32.587937 17207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 16:50:32.588114 17207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "58662c670a204a0d9fbf0403a4ea1822"
format_stamp: "Formatted at 2026-09-28 16:50:32 on dist-test-slave-wzcb"
server_key: "c3daa58289348f35e423e487feca1c9d"
server_key_iv: "2a8faf34d0730673507f3a8723ff658a"
server_key_version: "encryptionkey@0"
I20260928 16:50:32.588332 17207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:32.614406 17207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:32.619133 17207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:32.619457 17207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:32.641726 17207 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:33613
I20260928 16:50:32.641712 17284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:33613 every 8 connection(s)
I20260928 16:50:32.643216 17207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:50:32.643852 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 17207
I20260928 16:50:32.644081 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:50:32.644996 9588 external_mini_cluster.cc:1428] Setting key e9f08fa8a31ea51fce09ceadd4e036b7
I20260928 16:50:32.648540 17285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 16:50:32 dist-test-slave-wzcb krb5kdc[17194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM
I20260928 16:50:32.662518 17285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822: Bootstrap starting.
I20260928 16:50:32.667131 17285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:32.668740 17285 log.cc:824] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822: Log is configured to *not* fsync() on all Append() calls
I20260928 16:50:32.671087 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:32.647340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39860 (local address 127.9.93.62:33613)
0928 16:50:32.648216 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:32.648230 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:32.648276 (+ 46us) server_negotiation.cc:407] Connection header received
0928 16:50:32.650050 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:32.650074 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:32.650511 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:32.650859 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:32.652733 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:32.654179 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:32.656017 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:32.656728 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:32.656879 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:32.661777 (+ 4898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:32.661790 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:32.661814 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:32.661868 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:32.665853 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:32.666713 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:32.666724 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:32.666734 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:32.666992 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:32.667531 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:32.667536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:32.667540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:32.668348 (+ 808us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:32.668682 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:32.669119 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 16:50:32.669671 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":390,"threads_started":1}
I20260928 16:50:32.672541 17285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822: No bootstrap required, opened a new log
I20260928 16:50:32.680333 17285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58662c670a204a0d9fbf0403a4ea1822" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33613 } }
I20260928 16:50:32.680805 17285 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:32.680910 17285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58662c670a204a0d9fbf0403a4ea1822, State: Initialized, Role: FOLLOWER
I20260928 16:50:32.681521 17285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58662c670a204a0d9fbf0403a4ea1822" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33613 } }
I20260928 16:50:32.681754 17285 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:50:32.681892 17285 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:50:32.682062 17285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:32.684521 17285 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58662c670a204a0d9fbf0403a4ea1822" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33613 } }
I20260928 16:50:32.685122 17285 leader_election.cc:302] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58662c670a204a0d9fbf0403a4ea1822; no voters:
I20260928 16:50:32.685737 17285 leader_election.cc:288] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:50:32.685904 17290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:50:32.686684 17290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822 [term 1 LEADER]: Becoming Leader. State: Replica: 58662c670a204a0d9fbf0403a4ea1822, State: Running, Role: LEADER
I20260928 16:50:32.687400 17290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58662c670a204a0d9fbf0403a4ea1822" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33613 } }
I20260928 16:50:32.688026 17285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:50:32.690536 17291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58662c670a204a0d9fbf0403a4ea1822" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58662c670a204a0d9fbf0403a4ea1822" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33613 } } }
I20260928 16:50:32.691075 17291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822 [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:32.691072 17292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58662c670a204a0d9fbf0403a4ea1822. Latest consensus state: current_term: 1 leader_uuid: "58662c670a204a0d9fbf0403a4ea1822" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58662c670a204a0d9fbf0403a4ea1822" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33613 } } }
I20260928 16:50:32.691987 17292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822 [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:32.693390 17298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:50:32.699710 17298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:50:32.711158 17298 catalog_manager.cc:1347] Generated new cluster ID: fa2bd3cfeb4e45348f41dfa68777e6e6
I20260928 16:50:32.711349 17298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:50:32.730352 17298 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:50:32.731914 17298 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:50:32.741698 17298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822: Generated new TSK 0
I20260928 16:50:32.742708 17298 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.1.keytab.
Entry for principal oryx/127.9.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.1.keytab.
I20260928 16:50:32.818728 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33613
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.1.keytab
--principal=oryx/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:42491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:50:33.216048 17312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:33.216405 17312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:33.216586 17312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:33.224861 17312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:33.225008 17312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:33.225147 17312 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:50:33.225270 17312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:50:33.235999 17312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:42491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.1.keytab
--principal=oryx/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.93.62:33613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:33.237692 17312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:33.239183 17312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:33.253600 17320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:33.253671 17318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:33.253577 17317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:33.254567 17312 server_base.cc:1034] running on GCE node
I20260928 16:50:33.255241 17312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:33.256318 17312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:33.257570 17312 hybrid_clock.cc:648] HybridClock initialized: now 1790614233257473 us; error 64 us; skew 500 ppm
I20260928 16:50:33.257902 17312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:33 dist-test-slave-wzcb krb5kdc[17194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614233, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:33.261435 17312 init.cc:377] Logged in from keytab as oryx/127.9.93.1@KRBTEST.COM (short username oryx)
I20260928 16:50:33.263128 17312 webserver.cc:460] Webserver started at http://127.9.93.1:42973/ using document root <none> and password file <none>
I20260928 16:50:33.263953 17312 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:33.264067 17312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:33.264398 17312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:33.267774 17312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "0aaabc4fbfc74e0297fcd463c14d495b"
format_stamp: "Formatted at 2026-09-28 16:50:33 on dist-test-slave-wzcb"
server_key: "9432f96b7d87e9bb93d870a978ac867f"
server_key_iv: "bc96e6b648b476a70b6656af54ab9c1b"
server_key_version: "encryptionkey@0"
I20260928 16:50:33.268635 17312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "0aaabc4fbfc74e0297fcd463c14d495b"
format_stamp: "Formatted at 2026-09-28 16:50:33 on dist-test-slave-wzcb"
server_key: "9432f96b7d87e9bb93d870a978ac867f"
server_key_iv: "bc96e6b648b476a70b6656af54ab9c1b"
server_key_version: "encryptionkey@0"
I20260928 16:50:33.274255 17312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:50:33.278311 17327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:33.279892 17312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:50:33.280073 17312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "0aaabc4fbfc74e0297fcd463c14d495b"
format_stamp: "Formatted at 2026-09-28 16:50:33 on dist-test-slave-wzcb"
server_key: "9432f96b7d87e9bb93d870a978ac867f"
server_key_iv: "bc96e6b648b476a70b6656af54ab9c1b"
server_key_version: "encryptionkey@0"
I20260928 16:50:33.280328 17312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:33.298162 17312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:33.303076 17312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:33.303447 17312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:33.306703 17312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:33.306821 17312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:33.306988 17312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:33.307104 17312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 28 16:50:33 dist-test-slave-wzcb krb5kdc[17194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614233, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM
I20260928 16:50:33.331876 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:33.307190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:37697 (local address 127.9.93.62:33613)
0928 16:50:33.307792 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:33.307800 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:33.309221 (+ 1421us) server_negotiation.cc:407] Connection header received
0928 16:50:33.311574 (+ 2353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:33.311582 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:33.311687 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:33.311830 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:33.314411 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:33.315481 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:33.318213 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:33.318876 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:33.319039 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:33.324054 (+ 5015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:33.324064 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:33.324069 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:33.324131 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:33.327423 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:33.328806 (+ 1383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:33.328812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:33.328816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:33.329008 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:33.329570 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:33.329576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:33.329580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:33.329887 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:33.330035 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:33.331199 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0928 16:50:33.331492 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":354,"threads_started":1}
I20260928 16:50:33.333156 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:33.308082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33613 (local address 127.9.93.1:37697)
0928 16:50:33.309022 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:33.309069 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:33.311146 (+ 2077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:33.312117 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:33.312132 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:33.313130 (+ 998us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:33.314192 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:33.314227 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:33.315665 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:33.315669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:33.318022 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:33.318041 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:33.319237 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:33.319246 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:33.319513 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:33.320353 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:33.320381 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:33.323778 (+ 3397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:33.327613 (+ 3835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:33.327624 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:33.327651 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:33.328597 (+ 946us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:33.329155 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:33.329161 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:33.329164 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:33.329438 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:33.330026 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:33.330035 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:33.330639 (+ 604us) client_negotiation.cc:769] Sending connection context
0928 16:50:33.331067 (+ 428us) client_negotiation.cc:240] Negotiation successful
0928 16:50:33.331783 (+ 716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":436,"threads_started":1}
I20260928 16:50:33.354147 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:33.339858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:40463 (local address 127.9.93.62:33613)
0928 16:50:33.340319 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:33.340326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:33.340867 (+ 541us) server_negotiation.cc:407] Connection header received
0928 16:50:33.341057 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:33.341063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:33.341147 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:33.341232 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:33.342283 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:33.343020 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:33.344841 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:33.345459 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:33.345580 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:33.348222 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:33.348231 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:33.348237 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:33.348282 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:33.351236 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:33.352045 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:33.352051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:33.352055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:33.352203 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:33.352603 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:33.352608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:33.352612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:33.352918 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:33.353073 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:33.353506 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 16:50:33.353755 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308}
I20260928 16:50:33.354239 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:33.339910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33613 (local address 127.9.93.1:40463)
0928 16:50:33.340748 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:33.340770 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:33.340932 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:33.341279 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:33.341283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:33.341808 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:33.342135 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:33.342146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:33.343159 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:33.343165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:33.344680 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:33.344701 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:33.345686 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:33.345693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:33.345879 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:33.346701 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:33.346719 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:33.348000 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:33.351418 (+ 3418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:33.351425 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:33.351429 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:33.351920 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:33.352319 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:33.352328 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:33.352333 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:33.352489 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:33.353046 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:33.353051 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:33.353254 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 16:50:33.353381 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 16:50:33.353777 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":420,"threads_started":1}
I20260928 16:50:33.364872 17312 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:40001
I20260928 16:50:33.364892 17448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:40001 every 8 connection(s)
I20260928 16:50:33.366240 17312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:50:33.368573 17449 heartbeater.cc:347] Connected to a master server at 127.9.93.62:33613
I20260928 16:50:33.368801 17449 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:33.369585 17449 heartbeater.cc:511] Master 127.9.93.62:33613 requested a full tablet report, sending...
I20260928 16:50:33.369755 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 17312
I20260928 16:50:33.369961 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:50:33.370719 9588 external_mini_cluster.cc:1428] Setting key be18d34157adc391b9f25a835286ac55
I20260928 16:50:33.372072 17240 ts_manager.cc:194] Registered new tserver with Master: 0aaabc4fbfc74e0297fcd463c14d495b (127.9.93.1:40001)
I20260928 16:50:33.373924 17240 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.93.1@KRBTEST.COM'} at 127.9.93.1:37697
WARNING: no policy specified for oryx/127.9.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.2.keytab.
Entry for principal oryx/127.9.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.2.keytab.
I20260928 16:50:33.424664 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33613
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.2.keytab
--principal=oryx/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:42491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:50:33.790834 17453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:33.791199 17453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:33.791390 17453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:33.799822 17453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:33.799984 17453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:33.800139 17453 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:50:33.800267 17453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:50:33.811224 17453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:42491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.2.keytab
--principal=oryx/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.93.62:33613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:33.812912 17453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:33.814512 17453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:33.829613 17459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:33.829689 17458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:33.829718 17461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:33.830458 17453 server_base.cc:1034] running on GCE node
I20260928 16:50:33.831260 17453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:33.832548 17453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:33.833798 17453 hybrid_clock.cc:648] HybridClock initialized: now 1790614233833719 us; error 52 us; skew 500 ppm
I20260928 16:50:33.834164 17453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:33 dist-test-slave-wzcb krb5kdc[17194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614233, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:33.837837 17453 init.cc:377] Logged in from keytab as oryx/127.9.93.2@KRBTEST.COM (short username oryx)
I20260928 16:50:33.839864 17453 webserver.cc:460] Webserver started at http://127.9.93.2:45435/ using document root <none> and password file <none>
I20260928 16:50:33.840740 17453 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:33.840847 17453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:33.841183 17453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:33.844744 17453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "3d8a14c88c364c6bb93a15a06febc220"
format_stamp: "Formatted at 2026-09-28 16:50:33 on dist-test-slave-wzcb"
server_key: "5b923994855a4791d460dfd5e90061dc"
server_key_iv: "9f048902b28935503eecf5960da6db0b"
server_key_version: "encryptionkey@0"
I20260928 16:50:33.845647 17453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "3d8a14c88c364c6bb93a15a06febc220"
format_stamp: "Formatted at 2026-09-28 16:50:33 on dist-test-slave-wzcb"
server_key: "5b923994855a4791d460dfd5e90061dc"
server_key_iv: "9f048902b28935503eecf5960da6db0b"
server_key_version: "encryptionkey@0"
I20260928 16:50:33.852157 17453 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 16:50:33.856525 17468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:33.858306 17453 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:50:33.858529 17453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "3d8a14c88c364c6bb93a15a06febc220"
format_stamp: "Formatted at 2026-09-28 16:50:33 on dist-test-slave-wzcb"
server_key: "5b923994855a4791d460dfd5e90061dc"
server_key_iv: "9f048902b28935503eecf5960da6db0b"
server_key_version: "encryptionkey@0"
I20260928 16:50:33.858810 17453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:33.888192 17453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:33.893430 17453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:33.893770 17453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:33.897074 17453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:33.897176 17453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:33.897316 17453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:33.897383 17453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:50:33 dist-test-slave-wzcb krb5kdc[17194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614233, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM
I20260928 16:50:33.922653 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:33.897765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:48275 (local address 127.9.93.62:33613)
0928 16:50:33.898282 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:33.898288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:33.900270 (+ 1982us) server_negotiation.cc:407] Connection header received
0928 16:50:33.902599 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:33.902606 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:33.902712 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:33.902822 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:33.904796 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:33.905932 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:33.909335 (+ 3403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:33.909993 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:33.910146 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:33.915171 (+ 5025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:33.915180 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:33.915185 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:33.915238 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:33.918808 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:33.919892 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:33.919899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:33.919903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:33.920065 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:33.920601 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:33.920605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:33.920608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:33.920931 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:33.921078 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:33.921953 (+ 875us) server_negotiation.cc:299] Negotiation successful
0928 16:50:33.922230 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":280,"threads_started":1}
I20260928 16:50:33.924253 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:33.898874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33613 (local address 127.9.93.2:48275)
0928 16:50:33.900037 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:33.900113 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:33.902252 (+ 2139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:33.903131 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:33.903142 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:33.903875 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:33.904588 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:33.904610 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:33.906143 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:33.906149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:33.909059 (+ 2910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:33.909098 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:33.910297 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:33.910305 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:33.910544 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:33.911473 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:33.911516 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:33.914836 (+ 3320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:33.919018 (+ 4182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:33.919028 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:33.919049 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:33.919744 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:33.920194 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:33.920198 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:33.920201 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:33.920463 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:33.921079 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:33.921088 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:33.921537 (+ 449us) client_negotiation.cc:769] Sending connection context
0928 16:50:33.921818 (+ 281us) client_negotiation.cc:240] Negotiation successful
0928 16:50:33.922499 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"thread_start_us":591,"threads_started":1}
I20260928 16:50:33.948695 17548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:33.931463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33613 (local address 127.9.93.2:41991)
0928 16:50:33.932409 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:33.932428 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:33.932605 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:33.932997 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:33.933001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:33.933555 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:33.933913 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:33.933925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:33.935241 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:33.935247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:33.937054 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:33.937077 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:33.938474 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:33.938480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:33.938632 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:33.939409 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:33.939442 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:33.940684 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:33.945026 (+ 4342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:33.945034 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:33.945040 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:33.945888 (+ 848us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:33.946426 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:33.946432 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:33.946436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:33.946672 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:33.947422 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:33.947427 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:33.947696 (+ 269us) client_negotiation.cc:769] Sending connection context
0928 16:50:33.947890 (+ 194us) client_negotiation.cc:240] Negotiation successful
0928 16:50:33.948296 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":524,"threads_started":1}
I20260928 16:50:33.948788 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:33.931507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:41991 (local address 127.9.93.62:33613)
0928 16:50:33.931725 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:33.931733 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:33.932535 (+ 802us) server_negotiation.cc:407] Connection header received
0928 16:50:33.932757 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:33.932761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:33.932838 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:33.932936 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:33.934090 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:33.935038 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:33.937383 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:33.938191 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:33.938379 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:33.941026 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:33.941032 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:33.941035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:33.941085 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:33.944824 (+ 3739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:33.946094 (+ 1270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:33.946102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:33.946105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:33.946270 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:33.946859 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:33.946863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:33.946866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:33.947271 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:33.947472 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:33.948075 (+ 603us) server_negotiation.cc:299] Negotiation successful
0928 16:50:33.948390 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 16:50:33.955150 17453 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:33729
I20260928 16:50:33.955169 17589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:33729 every 8 connection(s)
I20260928 16:50:33.956547 17453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:50:33.958866 17590 heartbeater.cc:347] Connected to a master server at 127.9.93.62:33613
I20260928 16:50:33.959090 17590 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:33.959880 17590 heartbeater.cc:511] Master 127.9.93.62:33613 requested a full tablet report, sending...
I20260928 16:50:33.960728 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 17453
I20260928 16:50:33.960913 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:50:33.961431 9588 external_mini_cluster.cc:1428] Setting key 71b813beaf706dbbfe4af5ffc32a4bf6
I20260928 16:50:33.961477 17240 ts_manager.cc:194] Registered new tserver with Master: 3d8a14c88c364c6bb93a15a06febc220 (127.9.93.2:33729)
I20260928 16:50:33.962500 17240 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.93.2@KRBTEST.COM'} at 127.9.93.2:48275
WARNING: no policy specified for oryx/127.9.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.3.keytab.
Entry for principal oryx/127.9.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.3.keytab.
I20260928 16:50:34.016271 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33613
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.3.keytab
--principal=oryx/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:42491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:50:34.377367 17449 heartbeater.cc:503] Master 127.9.93.62:33613 was elected leader, sending a full tablet report...
W20260928 16:50:34.389894 17594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:34.390232 17594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:34.390352 17594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:34.398592 17594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:34.398732 17594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:34.398824 17594 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:50:34.398937 17594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:50:34.409940 17594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:42491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.3.keytab
--principal=oryx/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.93.62:33613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:34.411547 17594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:34.413177 17594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:34.428087 17599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:34.428162 17600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:34.428189 17602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:34.429107 17594 server_base.cc:1034] running on GCE node
I20260928 16:50:34.429814 17594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:34.430963 17594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:34.432229 17594 hybrid_clock.cc:648] HybridClock initialized: now 1790614234432114 us; error 75 us; skew 500 ppm
I20260928 16:50:34.432574 17594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:34 dist-test-slave-wzcb krb5kdc[17194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614234, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:34.436239 17594 init.cc:377] Logged in from keytab as oryx/127.9.93.3@KRBTEST.COM (short username oryx)
I20260928 16:50:34.437964 17594 webserver.cc:460] Webserver started at http://127.9.93.3:37581/ using document root <none> and password file <none>
I20260928 16:50:34.438782 17594 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:34.438885 17594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:34.439208 17594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:34.442700 17594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "b0f38fc994554643bf99c195029e1557"
format_stamp: "Formatted at 2026-09-28 16:50:34 on dist-test-slave-wzcb"
server_key: "20626e9306e544cbf132eaeba3f0aab7"
server_key_iv: "5cde51826d522d2cc1fefd8d2f20ba1f"
server_key_version: "encryptionkey@0"
I20260928 16:50:34.443677 17594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "b0f38fc994554643bf99c195029e1557"
format_stamp: "Formatted at 2026-09-28 16:50:34 on dist-test-slave-wzcb"
server_key: "20626e9306e544cbf132eaeba3f0aab7"
server_key_iv: "5cde51826d522d2cc1fefd8d2f20ba1f"
server_key_version: "encryptionkey@0"
I20260928 16:50:34.449833 17594 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:50:34.454094 17609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:34.456229 17594 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 16:50:34.456421 17594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "b0f38fc994554643bf99c195029e1557"
format_stamp: "Formatted at 2026-09-28 16:50:34 on dist-test-slave-wzcb"
server_key: "20626e9306e544cbf132eaeba3f0aab7"
server_key_iv: "5cde51826d522d2cc1fefd8d2f20ba1f"
server_key_version: "encryptionkey@0"
I20260928 16:50:34.456681 17594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:34.474069 17594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:34.479444 17594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:34.479759 17594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:34.482870 17594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:34.482991 17594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:34.483139 17594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:34.483217 17594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:50:34 dist-test-slave-wzcb krb5kdc[17194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614234, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM
I20260928 16:50:34.511197 17618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:34.483853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:52763 (local address 127.9.93.62:33613)
0928 16:50:34.484418 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:34.484424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:34.486544 (+ 2120us) server_negotiation.cc:407] Connection header received
0928 16:50:34.489184 (+ 2640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:34.489190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:34.489289 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:34.489393 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:34.491830 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.493054 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.496331 (+ 3277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.496781 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.496901 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:34.503171 (+ 6270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:34.503180 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:34.503185 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:34.503243 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:34.506801 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:34.507825 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:34.507834 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:34.507837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:34.508045 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:34.508726 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:34.508732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:34.508736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:34.509113 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:34.509329 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:34.510685 (+ 1356us) server_negotiation.cc:299] Negotiation successful
0928 16:50:34.510888 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":245,"threads_started":1}
I20260928 16:50:34.513548 17619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:34.485298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33613 (local address 127.9.93.3:52763)
0928 16:50:34.486289 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:34.486352 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:34.488857 (+ 2505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:34.489704 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:34.489714 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:34.490633 (+ 919us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:34.491634 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:34.491659 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.493240 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.493250 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:34.496107 (+ 2857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:34.496132 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.497071 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.497080 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:34.497472 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:34.498462 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:34.498511 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:34.502711 (+ 4200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:34.506965 (+ 4254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:34.506977 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:34.507001 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:34.507647 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:34.508205 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:34.508210 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:34.508214 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:34.508528 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:34.509327 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:34.509340 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:34.510067 (+ 727us) client_negotiation.cc:769] Sending connection context
0928 16:50:34.510567 (+ 500us) client_negotiation.cc:240] Negotiation successful
0928 16:50:34.511371 (+ 804us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":448,"threads_started":1}
I20260928 16:50:34.528515 17594 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:37881
I20260928 16:50:34.528580 17729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:37881 every 8 connection(s)
I20260928 16:50:34.530154 17594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:50:34.532235 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 17594
I20260928 16:50:34.532423 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:50:34.533011 17730 heartbeater.cc:347] Connected to a master server at 127.9.93.62:33613
I20260928 16:50:34.533073 9588 external_mini_cluster.cc:1428] Setting key 0a4844b92ccf6ee1db18c0c189da809d
I20260928 16:50:34.533298 17730 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:34.534343 17730 heartbeater.cc:511] Master 127.9.93.62:33613 requested a full tablet report, sending...
I20260928 16:50:34.536110 17240 ts_manager.cc:194] Registered new tserver with Master: b0f38fc994554643bf99c195029e1557 (127.9.93.3:37881)
I20260928 16:50:34.536554 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:34.520772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33613 (local address 127.9.93.3:48087)
0928 16:50:34.521458 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:34.521479 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:34.521645 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:34.522047 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:34.522051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:34.522522 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:34.522847 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:34.522858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.524414 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.524420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:34.526169 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:34.526190 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.527274 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.527281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:34.527493 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:34.528295 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:34.528324 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:34.529339 (+ 1015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:34.533351 (+ 4012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:34.533357 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:34.533361 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:34.533936 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:34.534434 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:34.534439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:34.534443 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:34.534631 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:34.535243 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:34.535248 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:34.535493 (+ 245us) client_negotiation.cc:769] Sending connection context
0928 16:50:34.535656 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 16:50:34.536088 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":465,"threads_started":1}
I20260928 16:50:34.537192 17240 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.93.3@KRBTEST.COM'} at 127.9.93.3:52763
I20260928 16:50:34.537492 17618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:34.521168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:48087 (local address 127.9.93.62:33613)
0928 16:50:34.521397 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:34.521405 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:34.521569 (+ 164us) server_negotiation.cc:407] Connection header received
0928 16:50:34.521783 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:34.521791 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:34.521884 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:34.522019 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:34.522993 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.524218 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.526334 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.527031 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.527217 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:34.529576 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:34.529584 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:34.529588 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:34.529633 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:34.533201 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:34.534103 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:34.534109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:34.534113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:34.534309 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:34.534750 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:34.534756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:34.534760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:34.535121 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:34.535268 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:34.536906 (+ 1638us) server_negotiation.cc:299] Negotiation successful
0928 16:50:34.537157 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 16:50:34.549782 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 16:50:34 dist-test-slave-wzcb krb5kdc[17194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614234, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 16:50:34 dist-test-slave-wzcb krb5kdc[17194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:34 dist-test-slave-wzcb krb5kdc[17194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 16:50:34.604913 17618 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:34.593749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.9.93.62:33613)
0928 16:50:34.593926 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:34.593931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:34.594231 (+ 300us) server_negotiation.cc:407] Connection header received
0928 16:50:34.594408 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:34.594414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:34.594534 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:34.594636 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:34.596178 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.596978 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.598276 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.598734 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.598840 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:34.604640 (+ 5800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39866: BlockingRecv error: recv got EOF from 127.0.0.1:39866 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 16:50:34 dist-test-slave-wzcb krb5kdc[17194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614234, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM
I20260928 16:50:34.622290 17618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:34.608061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39870 (local address 127.9.93.62:33613)
0928 16:50:34.608248 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:34.608255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:34.608566 (+ 311us) server_negotiation.cc:407] Connection header received
0928 16:50:34.608753 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:34.608757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:34.608824 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:34.608909 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:34.610060 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.610794 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.612123 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.612592 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.612717 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:34.616691 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:34.616697 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:34.616700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:34.616742 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:34.619560 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:34.620305 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:34.620310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:34.620312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:34.620509 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:34.620863 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:34.620868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:34.620870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:34.621181 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:34.621409 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:34.621836 (+ 427us) server_negotiation.cc:299] Negotiation successful
0928 16:50:34.622039 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:50:34.626536 17240 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:50:34.634161 17240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:50:34.661559 17749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:34.648815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:40001 (local address 127.0.0.1:33490)
0928 16:50:34.649793 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:34.649843 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:34.650038 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:34.651952 (+ 1914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:34.651962 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:34.652001 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:50:34.652542 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:34.652556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.654973 (+ 2417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.654980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:34.657908 (+ 2928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:34.657934 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.660446 (+ 2512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.660454 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:34.660744 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:34.660823 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 16:50:34.660940 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 16:50:34.661158 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":520,"threads_started":1}
I20260928 16:50:34.663069 17750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:34.649140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33490 (local address 127.9.93.1:40001)
0928 16:50:34.651100 (+ 1960us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:34.651114 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:34.651152 (+ 38us) server_negotiation.cc:407] Connection header received
0928 16:50:34.651280 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:34.651290 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:34.651668 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:34.652802 (+ 1134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:50:34.653061 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.654777 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.658270 (+ 3493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.660172 (+ 1902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.662147 (+ 1975us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:34.662426 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:34.662511 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 16:50:34.662739 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1769,"thread_start_us":1612,"threads_started":1}
I20260928 16:50:34.668764 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:34.650493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:33729 (local address 127.0.0.1:35056)
0928 16:50:34.660373 (+ 9880us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:34.660402 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:34.660659 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:34.661753 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:34.661759 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:34.661786 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:50:34.662249 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:34.662263 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.664255 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.664260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:34.666268 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:34.666284 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.667873 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.667880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:34.668087 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:34.668129 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 16:50:34.668223 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 16:50:34.668369 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9723,"thread_start_us":9528,"threads_started":1}
I20260928 16:50:34.669680 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:34.650824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35056 (local address 127.9.93.2:33729)
0928 16:50:34.659967 (+ 9143us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:34.659981 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:34.660512 (+ 531us) server_negotiation.cc:407] Connection header received
0928 16:50:34.660888 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:34.660900 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:34.661268 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:34.661471 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:50:34.662459 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.664060 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.666448 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.667638 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.667780 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:34.668023 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:34.669136 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0928 16:50:34.669348 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8856,"thread_start_us":8651,"threads_started":1}
I20260928 16:50:34.670490 17378 tablet_service.cc:1467] Processing CreateTablet for tablet da804037d41c40e98a88cf76cdce2072 (DEFAULT_TABLE table=test-table [id=6d7ae94c6843497a85a9ebf17eb3ee91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:50:34.671875 17749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:34.664359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:37881 (local address 127.0.0.1:37130)
0928 16:50:34.664564 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:34.664585 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:34.664734 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:34.665462 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:34.665468 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:34.665495 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:50:34.665740 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:34.665750 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.667668 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.667673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:34.669496 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:34.669512 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.671083 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.671090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:34.671280 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:34.671352 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 16:50:34.671442 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 16:50:34.671607 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 16:50:34.672039 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:34.660946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37130 (local address 127.9.93.3:37881)
0928 16:50:34.664279 (+ 3333us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:34.664288 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:34.664708 (+ 420us) server_negotiation.cc:407] Connection header received
0928 16:50:34.664909 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:34.664918 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:34.665238 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:34.665481 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:50:34.665959 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.667482 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.669690 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.670906 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.671036 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:34.671333 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:34.671500 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 16:50:34.671690 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3137,"thread_start_us":2931,"threads_started":1}
I20260928 16:50:34.672641 17378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da804037d41c40e98a88cf76cdce2072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:34.674444 17519 tablet_service.cc:1467] Processing CreateTablet for tablet da804037d41c40e98a88cf76cdce2072 (DEFAULT_TABLE table=test-table [id=6d7ae94c6843497a85a9ebf17eb3ee91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:50:34.676201 17519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da804037d41c40e98a88cf76cdce2072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:34.675632 17660 tablet_service.cc:1467] Processing CreateTablet for tablet da804037d41c40e98a88cf76cdce2072 (DEFAULT_TABLE table=test-table [id=6d7ae94c6843497a85a9ebf17eb3ee91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:50:34.677093 17660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da804037d41c40e98a88cf76cdce2072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:34.690512 17754 tablet_bootstrap.cc:492] T da804037d41c40e98a88cf76cdce2072 P 0aaabc4fbfc74e0297fcd463c14d495b: Bootstrap starting.
I20260928 16:50:34.693084 17755 tablet_bootstrap.cc:492] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557: Bootstrap starting.
I20260928 16:50:34.695545 17754 tablet_bootstrap.cc:654] T da804037d41c40e98a88cf76cdce2072 P 0aaabc4fbfc74e0297fcd463c14d495b: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:34.697047 17754 log.cc:824] T da804037d41c40e98a88cf76cdce2072 P 0aaabc4fbfc74e0297fcd463c14d495b: Log is configured to *not* fsync() on all Append() calls
I20260928 16:50:34.698696 17755 tablet_bootstrap.cc:654] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:34.698805 17756 tablet_bootstrap.cc:492] T da804037d41c40e98a88cf76cdce2072 P 3d8a14c88c364c6bb93a15a06febc220: Bootstrap starting.
I20260928 16:50:34.700424 17755 log.cc:824] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557: Log is configured to *not* fsync() on all Append() calls
I20260928 16:50:34.704382 17756 tablet_bootstrap.cc:654] T da804037d41c40e98a88cf76cdce2072 P 3d8a14c88c364c6bb93a15a06febc220: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:34.704644 17755 tablet_bootstrap.cc:492] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557: No bootstrap required, opened a new log
I20260928 16:50:34.704694 17754 tablet_bootstrap.cc:492] T da804037d41c40e98a88cf76cdce2072 P 0aaabc4fbfc74e0297fcd463c14d495b: No bootstrap required, opened a new log
I20260928 16:50:34.705075 17755 ts_tablet_manager.cc:1397] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260928 16:50:34.705034 17754 ts_tablet_manager.cc:1397] T da804037d41c40e98a88cf76cdce2072 P 0aaabc4fbfc74e0297fcd463c14d495b: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260928 16:50:34.706094 17756 log.cc:824] T da804037d41c40e98a88cf76cdce2072 P 3d8a14c88c364c6bb93a15a06febc220: Log is configured to *not* fsync() on all Append() calls
I20260928 16:50:34.709372 17756 tablet_bootstrap.cc:492] T da804037d41c40e98a88cf76cdce2072 P 3d8a14c88c364c6bb93a15a06febc220: No bootstrap required, opened a new log
I20260928 16:50:34.709718 17756 ts_tablet_manager.cc:1397] T da804037d41c40e98a88cf76cdce2072 P 3d8a14c88c364c6bb93a15a06febc220: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260928 16:50:34.716898 17755 raft_consensus.cc:348] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:34.716894 17754 raft_consensus.cc:348] T da804037d41c40e98a88cf76cdce2072 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:34.717455 17754 raft_consensus.cc:374] T da804037d41c40e98a88cf76cdce2072 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:34.717456 17755 raft_consensus.cc:374] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:34.717576 17755 raft_consensus.cc:729] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f38fc994554643bf99c195029e1557, State: Initialized, Role: FOLLOWER
I20260928 16:50:34.717643 17754 raft_consensus.cc:729] T da804037d41c40e98a88cf76cdce2072 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aaabc4fbfc74e0297fcd463c14d495b, State: Initialized, Role: FOLLOWER
I20260928 16:50:34.718350 17755 consensus_queue.cc:260] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:34.718348 17754 consensus_queue.cc:260] T da804037d41c40e98a88cf76cdce2072 P 0aaabc4fbfc74e0297fcd463c14d495b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:34.718415 17756 raft_consensus.cc:348] T da804037d41c40e98a88cf76cdce2072 P 3d8a14c88c364c6bb93a15a06febc220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:34.718886 17756 raft_consensus.cc:374] T da804037d41c40e98a88cf76cdce2072 P 3d8a14c88c364c6bb93a15a06febc220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:34.719053 17756 raft_consensus.cc:729] T da804037d41c40e98a88cf76cdce2072 P 3d8a14c88c364c6bb93a15a06febc220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d8a14c88c364c6bb93a15a06febc220, State: Initialized, Role: FOLLOWER
I20260928 16:50:34.719993 17756 consensus_queue.cc:260] T da804037d41c40e98a88cf76cdce2072 P 3d8a14c88c364c6bb93a15a06febc220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:34.720278 17730 heartbeater.cc:503] Master 127.9.93.62:33613 was elected leader, sending a full tablet report...
I20260928 16:50:34.720579 17754 ts_tablet_manager.cc:1428] T da804037d41c40e98a88cf76cdce2072 P 0aaabc4fbfc74e0297fcd463c14d495b: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
I20260928 16:50:34.720613 17755 ts_tablet_manager.cc:1428] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260928 16:50:34.721617 17590 heartbeater.cc:503] Master 127.9.93.62:33613 was elected leader, sending a full tablet report...
I20260928 16:50:34.722522 17756 ts_tablet_manager.cc:1428] T da804037d41c40e98a88cf76cdce2072 P 3d8a14c88c364c6bb93a15a06febc220: Time spent starting tablet: real 0.013s user 0.003s sys 0.011s
W20260928 16:50:34.782212 17731 tablet.cc:2369] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:50:34.853123 17761 raft_consensus.cc:482] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:50:34.853521 17761 raft_consensus.cc:504] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:34.855556 17761 leader_election.cc:288] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0aaabc4fbfc74e0297fcd463c14d495b (127.9.93.1:40001), 3d8a14c88c364c6bb93a15a06febc220 (127.9.93.2:33729)
I20260928 16:50:34.861954 17750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:34.855973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:59817 (local address 127.9.93.1:40001)
0928 16:50:34.856255 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:34.856261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:34.856285 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:50:34.856470 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:34.856478 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:34.856637 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:34.856735 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:50:34.857362 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.858306 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.860114 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.861016 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.861150 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:34.861264 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:34.861534 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 16:50:34.861696 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260928 16:50:34.862107 17619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:34.855838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:40001 (local address 127.9.93.3:59817)
0928 16:50:34.856077 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:34.856101 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:34.856260 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:34.856937 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:34.856940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:34.856966 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:50:34.857167 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:34.857179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.858450 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.858456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:34.859939 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:34.859952 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.861169 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.861175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:34.861374 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:34.861401 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 16:50:34.861487 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 16:50:34.861636 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":36}
I20260928 16:50:34.862800 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:34.856739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:42007 (local address 127.9.93.2:33729)
0928 16:50:34.856976 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:34.856983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:34.857368 (+ 385us) server_negotiation.cc:407] Connection header received
0928 16:50:34.857452 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:34.857455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:34.857520 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:34.857597 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:50:34.858198 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.858935 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.860863 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.861807 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.861933 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:34.862049 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:34.862386 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 16:50:34.862522 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:50:34.862864 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:34.856272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:33729 (local address 127.9.93.3:42007)
0928 16:50:34.857065 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:34.857085 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:34.857238 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:34.857633 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:34.857638 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:34.857663 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:50:34.858015 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:34.858026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.859071 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.859077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:34.860707 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:34.860720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.862021 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.862028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:34.862207 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:34.862246 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 16:50:34.862328 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 16:50:34.862471 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":44,"thread_start_us":887,"threads_started":1}
I20260928 16:50:34.863453 17404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da804037d41c40e98a88cf76cdce2072" candidate_uuid: "b0f38fc994554643bf99c195029e1557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" is_pre_election: true
I20260928 16:50:34.864183 17404 raft_consensus.cc:2454] T da804037d41c40e98a88cf76cdce2072 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0f38fc994554643bf99c195029e1557 in term 0.
I20260928 16:50:34.864089 17540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da804037d41c40e98a88cf76cdce2072" candidate_uuid: "b0f38fc994554643bf99c195029e1557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d8a14c88c364c6bb93a15a06febc220" is_pre_election: true
I20260928 16:50:34.864737 17540 raft_consensus.cc:2454] T da804037d41c40e98a88cf76cdce2072 P 3d8a14c88c364c6bb93a15a06febc220 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0f38fc994554643bf99c195029e1557 in term 0.
I20260928 16:50:34.864956 17611 leader_election.cc:302] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aaabc4fbfc74e0297fcd463c14d495b, b0f38fc994554643bf99c195029e1557; no voters:
I20260928 16:50:34.865427 17761 raft_consensus.cc:2781] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:50:34.865581 17761 raft_consensus.cc:482] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:50:34.865697 17761 raft_consensus.cc:3037] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:34.868257 17761 raft_consensus.cc:504] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:34.868994 17761 leader_election.cc:288] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557 [CANDIDATE]: Term 1 election: Requested vote from peers 0aaabc4fbfc74e0297fcd463c14d495b (127.9.93.1:40001), 3d8a14c88c364c6bb93a15a06febc220 (127.9.93.2:33729)
W20260928 16:50:34.869181 17450 tablet.cc:2369] T da804037d41c40e98a88cf76cdce2072 P 0aaabc4fbfc74e0297fcd463c14d495b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:50:34.869580 17404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da804037d41c40e98a88cf76cdce2072" candidate_uuid: "b0f38fc994554643bf99c195029e1557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aaabc4fbfc74e0297fcd463c14d495b"
I20260928 16:50:34.869634 17540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da804037d41c40e98a88cf76cdce2072" candidate_uuid: "b0f38fc994554643bf99c195029e1557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d8a14c88c364c6bb93a15a06febc220"
I20260928 16:50:34.869823 17404 raft_consensus.cc:3037] T da804037d41c40e98a88cf76cdce2072 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:34.869884 17540 raft_consensus.cc:3037] T da804037d41c40e98a88cf76cdce2072 P 3d8a14c88c364c6bb93a15a06febc220 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:34.872376 17404 raft_consensus.cc:2454] T da804037d41c40e98a88cf76cdce2072 P 0aaabc4fbfc74e0297fcd463c14d495b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0f38fc994554643bf99c195029e1557 in term 1.
I20260928 16:50:34.872563 17540 raft_consensus.cc:2454] T da804037d41c40e98a88cf76cdce2072 P 3d8a14c88c364c6bb93a15a06febc220 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0f38fc994554643bf99c195029e1557 in term 1.
I20260928 16:50:34.872885 17611 leader_election.cc:302] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aaabc4fbfc74e0297fcd463c14d495b, b0f38fc994554643bf99c195029e1557; no voters:
I20260928 16:50:34.873286 17761 raft_consensus.cc:2781] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:50:34.873787 17761 raft_consensus.cc:686] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557 [term 1 LEADER]: Becoming Leader. State: Replica: b0f38fc994554643bf99c195029e1557, State: Running, Role: LEADER
I20260928 16:50:34.874326 17761 consensus_queue.cc:237] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:34.879979 17240 catalog_manager.cc:5495] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0f38fc994554643bf99c195029e1557 (127.9.93.3). New cstate: current_term: 1 leader_uuid: "b0f38fc994554643bf99c195029e1557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } health_report { overall_health: HEALTHY } } }
I20260928 16:50:34.917771 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:34.911834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37138 (local address 127.9.93.3:37881)
0928 16:50:34.912103 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:34.912109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:34.912128 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:50:34.912246 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:34.912250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:34.912352 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:34.912503 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:50:34.913018 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.913902 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.915320 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:34.915850 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:34.915956 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:34.916393 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:50:34.917140 (+ 747us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:50:34.917253 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:34.917350 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 16:50:34.917445 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260928 16:50:34.927124 17540 raft_consensus.cc:1261] T da804037d41c40e98a88cf76cdce2072 P 3d8a14c88c364c6bb93a15a06febc220 [term 1 FOLLOWER]: Refusing update from remote peer b0f38fc994554643bf99c195029e1557: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:50:34.927161 17404 raft_consensus.cc:1261] T da804037d41c40e98a88cf76cdce2072 P 0aaabc4fbfc74e0297fcd463c14d495b [term 1 FOLLOWER]: Refusing update from remote peer b0f38fc994554643bf99c195029e1557: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:50:34.928320 17765 consensus_queue.cc:1035] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:50:34.928794 17761 consensus_queue.cc:1035] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:50:34.940323 17768 mvcc.cc:205] Tried to move back new op lower bound from 7334355906249895936 to 7334355906049622016. Current Snapshot: MvccSnapshot[applied={T|T < 7334355906249895936}]
I20260928 16:50:34.959410 17240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:50:34.959767 17240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:50:34.965097 17240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822: Sending DeleteTablet for 3 replicas of tablet da804037d41c40e98a88cf76cdce2072
I20260928 16:50:34.966456 17378 tablet_service.cc:1514] Processing DeleteTablet for tablet da804037d41c40e98a88cf76cdce2072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:34 UTC) from {username='oryx', principal='oryx/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:33490
I20260928 16:50:34.966706 17519 tablet_service.cc:1514] Processing DeleteTablet for tablet da804037d41c40e98a88cf76cdce2072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:34 UTC) from {username='oryx', principal='oryx/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:35056
I20260928 16:50:34.970243 17660 tablet_service.cc:1514] Processing DeleteTablet for tablet da804037d41c40e98a88cf76cdce2072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:34 UTC) from {username='oryx', principal='oryx/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:37130
I20260928 16:50:34.977716 17780 tablet_replica.cc:330] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557: stopping tablet replica
I20260928 16:50:34.977793 17240 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 16:50:34.978334 17780 raft_consensus.cc:2229] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557 [term 1 LEADER]: Raft consensus shutting down.
W20260928 16:50:34.978995 17240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:50:34.979210 17780 raft_consensus.cc:2258] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:34.981140 17779 tablet_replica.cc:330] T da804037d41c40e98a88cf76cdce2072 P 3d8a14c88c364c6bb93a15a06febc220: stopping tablet replica
I20260928 16:50:34.981248 17778 tablet_replica.cc:330] T da804037d41c40e98a88cf76cdce2072 P 0aaabc4fbfc74e0297fcd463c14d495b: stopping tablet replica
I20260928 16:50:34.981752 17778 raft_consensus.cc:2229] T da804037d41c40e98a88cf76cdce2072 P 0aaabc4fbfc74e0297fcd463c14d495b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:50:34.982429 17778 raft_consensus.cc:2258] T da804037d41c40e98a88cf76cdce2072 P 0aaabc4fbfc74e0297fcd463c14d495b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:34.985150 17778 ts_tablet_manager.cc:1905] T da804037d41c40e98a88cf76cdce2072 P 0aaabc4fbfc74e0297fcd463c14d495b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:50:34.988225 17378 tablet_service.cc:1467] Processing CreateTablet for tablet f369a491339b4b4c85ef4111f7210290 (DEFAULT_TABLE table=test-table [id=8876c80efb764023b0314c6b02242e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:50:34.988862 17378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f369a491339b4b4c85ef4111f7210290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:34.990140 17660 tablet_service.cc:1467] Processing CreateTablet for tablet f369a491339b4b4c85ef4111f7210290 (DEFAULT_TABLE table=test-table [id=8876c80efb764023b0314c6b02242e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:50:34.990724 17660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f369a491339b4b4c85ef4111f7210290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:34.993904 17778 ts_tablet_manager.cc:1918] T da804037d41c40e98a88cf76cdce2072 P 0aaabc4fbfc74e0297fcd463c14d495b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:50:34.994117 17778 log.cc:1196] T da804037d41c40e98a88cf76cdce2072 P 0aaabc4fbfc74e0297fcd463c14d495b: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-0/wal/wals/da804037d41c40e98a88cf76cdce2072
I20260928 16:50:34.994855 17778 ts_tablet_manager.cc:1939] T da804037d41c40e98a88cf76cdce2072 P 0aaabc4fbfc74e0297fcd463c14d495b: Deleting consensus metadata
I20260928 16:50:34.996963 17754 tablet_bootstrap.cc:492] T f369a491339b4b4c85ef4111f7210290 P 0aaabc4fbfc74e0297fcd463c14d495b: Bootstrap starting.
I20260928 16:50:34.997149 17780 ts_tablet_manager.cc:1905] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:50:34.997572 17224 catalog_manager.cc:4841] TS 0aaabc4fbfc74e0297fcd463c14d495b (127.9.93.1:40001): tablet da804037d41c40e98a88cf76cdce2072 (table test-table [id=6d7ae94c6843497a85a9ebf17eb3ee91]) successfully deleted
I20260928 16:50:34.998961 17754 tablet_bootstrap.cc:654] T f369a491339b4b4c85ef4111f7210290 P 0aaabc4fbfc74e0297fcd463c14d495b: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:35.000067 17519 tablet_service.cc:1467] Processing CreateTablet for tablet f369a491339b4b4c85ef4111f7210290 (DEFAULT_TABLE table=test-table [id=8876c80efb764023b0314c6b02242e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:50:35.000742 17519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f369a491339b4b4c85ef4111f7210290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:35.006381 17779 raft_consensus.cc:2229] T da804037d41c40e98a88cf76cdce2072 P 3d8a14c88c364c6bb93a15a06febc220 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:50:35.007033 17779 raft_consensus.cc:2258] T da804037d41c40e98a88cf76cdce2072 P 3d8a14c88c364c6bb93a15a06febc220 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:35.007479 17780 ts_tablet_manager.cc:1918] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:50:35.007792 17780 log.cc:1196] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/da804037d41c40e98a88cf76cdce2072
I20260928 16:50:35.008667 17756 tablet_bootstrap.cc:492] T f369a491339b4b4c85ef4111f7210290 P 3d8a14c88c364c6bb93a15a06febc220: Bootstrap starting.
I20260928 16:50:35.009244 17780 ts_tablet_manager.cc:1939] T da804037d41c40e98a88cf76cdce2072 P b0f38fc994554643bf99c195029e1557: Deleting consensus metadata
I20260928 16:50:35.010603 17756 tablet_bootstrap.cc:654] T f369a491339b4b4c85ef4111f7210290 P 3d8a14c88c364c6bb93a15a06febc220: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:35.011602 17224 catalog_manager.cc:4841] TS b0f38fc994554643bf99c195029e1557 (127.9.93.3:37881): tablet da804037d41c40e98a88cf76cdce2072 (table test-table [id=6d7ae94c6843497a85a9ebf17eb3ee91]) successfully deleted
I20260928 16:50:35.012810 17754 tablet_bootstrap.cc:492] T f369a491339b4b4c85ef4111f7210290 P 0aaabc4fbfc74e0297fcd463c14d495b: No bootstrap required, opened a new log
I20260928 16:50:35.013005 17754 ts_tablet_manager.cc:1397] T f369a491339b4b4c85ef4111f7210290 P 0aaabc4fbfc74e0297fcd463c14d495b: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.007s
I20260928 16:50:35.014000 17754 raft_consensus.cc:348] T f369a491339b4b4c85ef4111f7210290 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:35.014299 17754 raft_consensus.cc:374] T f369a491339b4b4c85ef4111f7210290 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:35.014397 17754 raft_consensus.cc:729] T f369a491339b4b4c85ef4111f7210290 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aaabc4fbfc74e0297fcd463c14d495b, State: Initialized, Role: FOLLOWER
I20260928 16:50:35.014714 17754 consensus_queue.cc:260] T f369a491339b4b4c85ef4111f7210290 P 0aaabc4fbfc74e0297fcd463c14d495b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:35.015460 17754 ts_tablet_manager.cc:1428] T f369a491339b4b4c85ef4111f7210290 P 0aaabc4fbfc74e0297fcd463c14d495b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 16:50:35.017140 17755 tablet_bootstrap.cc:492] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557: Bootstrap starting.
I20260928 16:50:35.019125 17755 tablet_bootstrap.cc:654] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:35.021874 17779 ts_tablet_manager.cc:1905] T da804037d41c40e98a88cf76cdce2072 P 3d8a14c88c364c6bb93a15a06febc220: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:50:35.027252 17756 tablet_bootstrap.cc:492] T f369a491339b4b4c85ef4111f7210290 P 3d8a14c88c364c6bb93a15a06febc220: No bootstrap required, opened a new log
I20260928 16:50:35.027480 17756 ts_tablet_manager.cc:1397] T f369a491339b4b4c85ef4111f7210290 P 3d8a14c88c364c6bb93a15a06febc220: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.008s
I20260928 16:50:35.028393 17755 tablet_bootstrap.cc:492] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557: No bootstrap required, opened a new log
I20260928 16:50:35.028578 17755 ts_tablet_manager.cc:1397] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260928 16:50:35.028599 17756 raft_consensus.cc:348] T f369a491339b4b4c85ef4111f7210290 P 3d8a14c88c364c6bb93a15a06febc220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:35.028887 17756 raft_consensus.cc:374] T f369a491339b4b4c85ef4111f7210290 P 3d8a14c88c364c6bb93a15a06febc220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:35.029018 17756 raft_consensus.cc:729] T f369a491339b4b4c85ef4111f7210290 P 3d8a14c88c364c6bb93a15a06febc220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d8a14c88c364c6bb93a15a06febc220, State: Initialized, Role: FOLLOWER
I20260928 16:50:35.029394 17756 consensus_queue.cc:260] T f369a491339b4b4c85ef4111f7210290 P 3d8a14c88c364c6bb93a15a06febc220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:35.029631 17755 raft_consensus.cc:348] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:35.029888 17755 raft_consensus.cc:374] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:35.029980 17755 raft_consensus.cc:729] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f38fc994554643bf99c195029e1557, State: Initialized, Role: FOLLOWER
I20260928 16:50:35.030414 17779 ts_tablet_manager.cc:1918] T da804037d41c40e98a88cf76cdce2072 P 3d8a14c88c364c6bb93a15a06febc220: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:50:35.030287 17755 consensus_queue.cc:260] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:35.030604 17779 log.cc:1196] T da804037d41c40e98a88cf76cdce2072 P 3d8a14c88c364c6bb93a15a06febc220: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/da804037d41c40e98a88cf76cdce2072
I20260928 16:50:35.030977 17755 ts_tablet_manager.cc:1428] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 16:50:35.031265 17779 ts_tablet_manager.cc:1939] T da804037d41c40e98a88cf76cdce2072 P 3d8a14c88c364c6bb93a15a06febc220: Deleting consensus metadata
I20260928 16:50:35.033324 17756 ts_tablet_manager.cc:1428] T f369a491339b4b4c85ef4111f7210290 P 3d8a14c88c364c6bb93a15a06febc220: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260928 16:50:35.034107 17224 catalog_manager.cc:4841] TS 3d8a14c88c364c6bb93a15a06febc220 (127.9.93.2:33729): tablet da804037d41c40e98a88cf76cdce2072 (table test-table [id=6d7ae94c6843497a85a9ebf17eb3ee91]) successfully deleted
W20260928 16:50:35.210281 17591 tablet.cc:2369] T f369a491339b4b4c85ef4111f7210290 P 3d8a14c88c364c6bb93a15a06febc220: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:50:35.216253 17765 raft_consensus.cc:482] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:50:35.216600 17765 raft_consensus.cc:504] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:35.217427 17765 leader_election.cc:288] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0aaabc4fbfc74e0297fcd463c14d495b (127.9.93.1:40001), 3d8a14c88c364c6bb93a15a06febc220 (127.9.93.2:33729)
I20260928 16:50:35.218281 17404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f369a491339b4b4c85ef4111f7210290" candidate_uuid: "b0f38fc994554643bf99c195029e1557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" is_pre_election: true
I20260928 16:50:35.218289 17540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f369a491339b4b4c85ef4111f7210290" candidate_uuid: "b0f38fc994554643bf99c195029e1557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d8a14c88c364c6bb93a15a06febc220" is_pre_election: true
I20260928 16:50:35.218595 17404 raft_consensus.cc:2454] T f369a491339b4b4c85ef4111f7210290 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0f38fc994554643bf99c195029e1557 in term 0.
I20260928 16:50:35.218595 17540 raft_consensus.cc:2454] T f369a491339b4b4c85ef4111f7210290 P 3d8a14c88c364c6bb93a15a06febc220 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0f38fc994554643bf99c195029e1557 in term 0.
I20260928 16:50:35.219082 17611 leader_election.cc:302] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8a14c88c364c6bb93a15a06febc220, b0f38fc994554643bf99c195029e1557; no voters:
I20260928 16:50:35.219410 17765 raft_consensus.cc:2781] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:50:35.219532 17765 raft_consensus.cc:482] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:50:35.219652 17765 raft_consensus.cc:3037] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:35.222122 17765 raft_consensus.cc:504] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:35.222800 17765 leader_election.cc:288] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557 [CANDIDATE]: Term 1 election: Requested vote from peers 0aaabc4fbfc74e0297fcd463c14d495b (127.9.93.1:40001), 3d8a14c88c364c6bb93a15a06febc220 (127.9.93.2:33729)
I20260928 16:50:35.223513 17540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f369a491339b4b4c85ef4111f7210290" candidate_uuid: "b0f38fc994554643bf99c195029e1557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d8a14c88c364c6bb93a15a06febc220"
I20260928 16:50:35.223765 17540 raft_consensus.cc:3037] T f369a491339b4b4c85ef4111f7210290 P 3d8a14c88c364c6bb93a15a06febc220 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:35.226122 17540 raft_consensus.cc:2454] T f369a491339b4b4c85ef4111f7210290 P 3d8a14c88c364c6bb93a15a06febc220 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0f38fc994554643bf99c195029e1557 in term 1.
I20260928 16:50:35.226401 17404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f369a491339b4b4c85ef4111f7210290" candidate_uuid: "b0f38fc994554643bf99c195029e1557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aaabc4fbfc74e0297fcd463c14d495b"
I20260928 16:50:35.226670 17404 raft_consensus.cc:3037] T f369a491339b4b4c85ef4111f7210290 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:35.226816 17611 leader_election.cc:302] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8a14c88c364c6bb93a15a06febc220, b0f38fc994554643bf99c195029e1557; no voters:
I20260928 16:50:35.227207 17765 raft_consensus.cc:2781] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:50:35.227419 17765 raft_consensus.cc:686] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557 [term 1 LEADER]: Becoming Leader. State: Replica: b0f38fc994554643bf99c195029e1557, State: Running, Role: LEADER
I20260928 16:50:35.227721 17765 consensus_queue.cc:237] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:35.229997 17404 raft_consensus.cc:2454] T f369a491339b4b4c85ef4111f7210290 P 0aaabc4fbfc74e0297fcd463c14d495b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0f38fc994554643bf99c195029e1557 in term 1.
I20260928 16:50:35.232925 17239 catalog_manager.cc:5495] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0f38fc994554643bf99c195029e1557 (127.9.93.3). New cstate: current_term: 1 leader_uuid: "b0f38fc994554643bf99c195029e1557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } health_report { overall_health: HEALTHY } } }
Sep 28 16:50:35 dist-test-slave-wzcb krb5kdc[17194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614232, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.62@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM
I20260928 16:50:35.304883 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:35.282163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33613 (local address 127.0.0.1:39874)
0928 16:50:35.283050 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:35.283067 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:35.283230 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:35.283731 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:35.283736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:35.284826 (+ 1090us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:35.285124 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:35.285134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.286725 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.286730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:35.287849 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:35.287865 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.288625 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.288631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:35.288810 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:35.295053 (+ 6243us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:35.295078 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:35.298168 (+ 3090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:35.301992 (+ 3824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:35.302013 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:35.302023 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:35.302518 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:35.302915 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:35.302920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:35.302924 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:35.303080 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:35.303728 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:35.303738 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:35.303916 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 16:50:35.304078 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 16:50:35.304397 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"thread_start_us":643,"threads_started":1}
I20260928 16:50:35.304910 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:35.282136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39874 (local address 127.9.93.62:33613)
0928 16:50:35.282948 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:35.282955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:35.283159 (+ 204us) server_negotiation.cc:407] Connection header received
0928 16:50:35.283405 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:35.283413 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:35.283501 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:35.283669 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:35.285309 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.286544 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.288018 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.288454 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.288572 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:35.298500 (+ 9928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:35.298507 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:35.298510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:35.298552 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:35.301820 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:35.302660 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:35.302665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:35.302668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:35.302801 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:35.303218 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:35.303222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:35.303224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:35.303582 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:35.303728 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:35.304196 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 16:50:35.304549 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":134,"server-negotiator.queue_time_us":627,"thread_start_us":398,"threads_started":1}
I20260928 16:50:35.311184 17239 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:39874:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 16:50:35.312425 17239 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:50:35.320662 17378 tablet_service.cc:1467] Processing CreateTablet for tablet 80b1d8797951439ba8ddb8637cb65c27 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1de5d55bb4af4ed7ae86eeaae17ef6b6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:50:35.320662 17519 tablet_service.cc:1467] Processing CreateTablet for tablet 80b1d8797951439ba8ddb8637cb65c27 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1de5d55bb4af4ed7ae86eeaae17ef6b6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:50:35.320912 17660 tablet_service.cc:1467] Processing CreateTablet for tablet 80b1d8797951439ba8ddb8637cb65c27 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1de5d55bb4af4ed7ae86eeaae17ef6b6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:50:35.321383 17519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b1d8797951439ba8ddb8637cb65c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:35.321382 17378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b1d8797951439ba8ddb8637cb65c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:35.321662 17660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b1d8797951439ba8ddb8637cb65c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:35.329794 17756 tablet_bootstrap.cc:492] T 80b1d8797951439ba8ddb8637cb65c27 P 3d8a14c88c364c6bb93a15a06febc220: Bootstrap starting.
I20260928 16:50:35.330444 17755 tablet_bootstrap.cc:492] T 80b1d8797951439ba8ddb8637cb65c27 P b0f38fc994554643bf99c195029e1557: Bootstrap starting.
I20260928 16:50:35.332139 17755 tablet_bootstrap.cc:654] T 80b1d8797951439ba8ddb8637cb65c27 P b0f38fc994554643bf99c195029e1557: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:35.332131 17756 tablet_bootstrap.cc:654] T 80b1d8797951439ba8ddb8637cb65c27 P 3d8a14c88c364c6bb93a15a06febc220: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:35.333421 17754 tablet_bootstrap.cc:492] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b: Bootstrap starting.
I20260928 16:50:35.334534 17755 tablet_bootstrap.cc:492] T 80b1d8797951439ba8ddb8637cb65c27 P b0f38fc994554643bf99c195029e1557: No bootstrap required, opened a new log
I20260928 16:50:35.334739 17755 ts_tablet_manager.cc:1397] T 80b1d8797951439ba8ddb8637cb65c27 P b0f38fc994554643bf99c195029e1557: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260928 16:50:35.335559 17754 tablet_bootstrap.cc:654] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:35.335606 17755 raft_consensus.cc:348] T 80b1d8797951439ba8ddb8637cb65c27 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:35.335906 17755 raft_consensus.cc:374] T 80b1d8797951439ba8ddb8637cb65c27 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:35.336004 17755 raft_consensus.cc:729] T 80b1d8797951439ba8ddb8637cb65c27 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f38fc994554643bf99c195029e1557, State: Initialized, Role: FOLLOWER
I20260928 16:50:35.336282 17755 consensus_queue.cc:260] T 80b1d8797951439ba8ddb8637cb65c27 P b0f38fc994554643bf99c195029e1557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:35.336848 17755 ts_tablet_manager.cc:1428] T 80b1d8797951439ba8ddb8637cb65c27 P b0f38fc994554643bf99c195029e1557: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 16:50:35.337256 17765 tablet_replica.cc:439] T 80b1d8797951439ba8ddb8637cb65c27 P b0f38fc994554643bf99c195029e1557: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } } }
I20260928 16:50:35.337590 17765 tablet_replica.cc:442] T 80b1d8797951439ba8ddb8637cb65c27 P b0f38fc994554643bf99c195029e1557: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:50:35.339372 17756 tablet_bootstrap.cc:492] T 80b1d8797951439ba8ddb8637cb65c27 P 3d8a14c88c364c6bb93a15a06febc220: No bootstrap required, opened a new log
I20260928 16:50:35.339612 17756 ts_tablet_manager.cc:1397] T 80b1d8797951439ba8ddb8637cb65c27 P 3d8a14c88c364c6bb93a15a06febc220: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260928 16:50:35.341328 17756 raft_consensus.cc:348] T 80b1d8797951439ba8ddb8637cb65c27 P 3d8a14c88c364c6bb93a15a06febc220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:35.341609 17756 raft_consensus.cc:374] T 80b1d8797951439ba8ddb8637cb65c27 P 3d8a14c88c364c6bb93a15a06febc220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:35.341696 17756 raft_consensus.cc:729] T 80b1d8797951439ba8ddb8637cb65c27 P 3d8a14c88c364c6bb93a15a06febc220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d8a14c88c364c6bb93a15a06febc220, State: Initialized, Role: FOLLOWER
I20260928 16:50:35.342016 17756 consensus_queue.cc:260] T 80b1d8797951439ba8ddb8637cb65c27 P 3d8a14c88c364c6bb93a15a06febc220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:35.342502 17763 tablet_replica.cc:439] T 80b1d8797951439ba8ddb8637cb65c27 P 3d8a14c88c364c6bb93a15a06febc220: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } } }
I20260928 16:50:35.342891 17763 tablet_replica.cc:442] T 80b1d8797951439ba8ddb8637cb65c27 P 3d8a14c88c364c6bb93a15a06febc220: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:50:35.343758 17756 ts_tablet_manager.cc:1428] T 80b1d8797951439ba8ddb8637cb65c27 P 3d8a14c88c364c6bb93a15a06febc220: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 16:50:35.349588 17754 tablet_bootstrap.cc:492] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b: No bootstrap required, opened a new log
I20260928 16:50:35.349823 17754 ts_tablet_manager.cc:1397] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.010s
I20260928 16:50:35.350728 17754 raft_consensus.cc:348] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:35.351025 17754 raft_consensus.cc:374] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:35.351136 17754 raft_consensus.cc:729] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aaabc4fbfc74e0297fcd463c14d495b, State: Initialized, Role: FOLLOWER
I20260928 16:50:35.351450 17754 consensus_queue.cc:260] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:35.352089 17762 tablet_replica.cc:439] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } } }
I20260928 16:50:35.352510 17762 tablet_replica.cc:442] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:50:35.353068 17754 ts_tablet_manager.cc:1428] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 16:50:35.619449 17765 consensus_queue.cc:1035] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:50:35.626406 17797 consensus_queue.cc:1035] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:50:35.659848 17762 raft_consensus.cc:482] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:50:35.660148 17762 raft_consensus.cc:504] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:35.662261 17762 leader_election.cc:288] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d8a14c88c364c6bb93a15a06febc220 (127.9.93.2:33729), b0f38fc994554643bf99c195029e1557 (127.9.93.3:37881)
I20260928 16:50:35.670169 17802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:35.662701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:32773 (local address 127.9.93.2:33729)
0928 16:50:35.663277 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:35.663283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:35.663332 (+ 49us) server_negotiation.cc:407] Connection header received
0928 16:50:35.663624 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:35.663631 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:35.663740 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:35.664559 (+ 819us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:50:35.664772 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.665815 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.668123 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.669035 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.669174 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:35.669289 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:35.669638 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 16:50:35.669822 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":276,"threads_started":1}
I20260928 16:50:35.670186 17804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:35.663629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:47577 (local address 127.9.93.3:37881)
0928 16:50:35.664386 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:35.664391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:35.664421 (+ 30us) server_negotiation.cc:407] Connection header received
0928 16:50:35.664494 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:35.664498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:35.664566 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:35.664656 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:50:35.665390 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.666228 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.668175 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.669040 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.669172 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:35.669288 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:35.669658 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 16:50:35.669825 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":462,"threads_started":1}
I20260928 16:50:35.670250 17801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:35.662588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:33729 (local address 127.9.93.1:32773)
0928 16:50:35.663189 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:35.663213 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:35.663435 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:35.663962 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:35.663968 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:35.664021 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:50:35.664401 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:35.664413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.666013 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.666021 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:35.667912 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:35.667936 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.669240 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.669250 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:35.669446 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:35.669485 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 16:50:35.669577 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 16:50:35.669796 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":141,"thread_start_us":268,"threads_started":1}
I20260928 16:50:35.670306 17803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:35.663410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:37881 (local address 127.9.93.1:47577)
0928 16:50:35.664093 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:35.664115 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:35.664245 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:35.664721 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:35.664726 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:35.664758 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:50:35.665159 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:35.665175 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.666376 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.666381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:35.667972 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:35.667988 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.669288 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.669294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:35.669486 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:35.669510 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:50:35.669591 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 16:50:35.669779 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":7,"thread_start_us":283,"threads_started":1}
I20260928 16:50:35.671411 17680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b1d8797951439ba8ddb8637cb65c27" candidate_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f38fc994554643bf99c195029e1557" is_pre_election: true
I20260928 16:50:35.671582 17540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b1d8797951439ba8ddb8637cb65c27" candidate_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d8a14c88c364c6bb93a15a06febc220" is_pre_election: true
I20260928 16:50:35.671878 17680 raft_consensus.cc:2454] T 80b1d8797951439ba8ddb8637cb65c27 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aaabc4fbfc74e0297fcd463c14d495b in term 0.
I20260928 16:50:35.671880 17540 raft_consensus.cc:2454] T 80b1d8797951439ba8ddb8637cb65c27 P 3d8a14c88c364c6bb93a15a06febc220 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aaabc4fbfc74e0297fcd463c14d495b in term 0.
I20260928 16:50:35.672487 17329 leader_election.cc:302] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aaabc4fbfc74e0297fcd463c14d495b, 3d8a14c88c364c6bb93a15a06febc220; no voters:
I20260928 16:50:35.672967 17762 raft_consensus.cc:2781] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:50:35.673136 17762 raft_consensus.cc:482] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:50:35.673241 17762 raft_consensus.cc:3037] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:35.675869 17762 raft_consensus.cc:504] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:35.676780 17762 leader_election.cc:288] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b [CANDIDATE]: Term 1 election: Requested vote from peers 3d8a14c88c364c6bb93a15a06febc220 (127.9.93.2:33729), b0f38fc994554643bf99c195029e1557 (127.9.93.3:37881)
I20260928 16:50:35.677551 17540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b1d8797951439ba8ddb8637cb65c27" candidate_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d8a14c88c364c6bb93a15a06febc220"
I20260928 16:50:35.677583 17680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b1d8797951439ba8ddb8637cb65c27" candidate_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f38fc994554643bf99c195029e1557"
I20260928 16:50:35.677821 17540 raft_consensus.cc:3037] T 80b1d8797951439ba8ddb8637cb65c27 P 3d8a14c88c364c6bb93a15a06febc220 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:35.677841 17680 raft_consensus.cc:3037] T 80b1d8797951439ba8ddb8637cb65c27 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:35.680323 17540 raft_consensus.cc:2454] T 80b1d8797951439ba8ddb8637cb65c27 P 3d8a14c88c364c6bb93a15a06febc220 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aaabc4fbfc74e0297fcd463c14d495b in term 1.
I20260928 16:50:35.680339 17680 raft_consensus.cc:2454] T 80b1d8797951439ba8ddb8637cb65c27 P b0f38fc994554643bf99c195029e1557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aaabc4fbfc74e0297fcd463c14d495b in term 1.
I20260928 16:50:35.680953 17329 leader_election.cc:302] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aaabc4fbfc74e0297fcd463c14d495b, b0f38fc994554643bf99c195029e1557; no voters:
I20260928 16:50:35.681345 17762 raft_consensus.cc:2781] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:50:35.681934 17762 raft_consensus.cc:686] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b [term 1 LEADER]: Becoming Leader. State: Replica: 0aaabc4fbfc74e0297fcd463c14d495b, State: Running, Role: LEADER
I20260928 16:50:35.682621 17762 consensus_queue.cc:237] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:35.685242 17805 tablet_replica.cc:439] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b: TxnStatusTablet state changed. Reason: New leader 0aaabc4fbfc74e0297fcd463c14d495b. Latest consensus state: current_term: 1 leader_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } } }
I20260928 16:50:35.685725 17805 tablet_replica.cc:442] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:50:35.686580 17807 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 16:50:35.686870 17807 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 16:50:35.686808 17240 catalog_manager.cc:5495] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0aaabc4fbfc74e0297fcd463c14d495b (127.9.93.1). New cstate: current_term: 1 leader_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } health_report { overall_health: UNKNOWN } } }
I20260928 16:50:35.724862 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:35.716392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:40001 (local address 127.0.0.1:33498)
0928 16:50:35.716790 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:35.716815 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:35.717005 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:35.718297 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:35.718302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:35.718315 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:50:35.718511 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:35.718521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.720324 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.720330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:35.721449 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:35.721466 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.722597 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.722602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:35.722797 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:35.722876 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:50:35.724298 (+ 1422us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:50:35.724341 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 16:50:35.724421 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 16:50:35.724592 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 16:50:35.725021 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:35.716726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33498 (local address 127.9.93.1:40001)
0928 16:50:35.717681 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:35.717687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:35.717719 (+ 32us) server_negotiation.cc:407] Connection header received
0928 16:50:35.717852 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:35.717866 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:35.717998 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:35.718242 (+ 244us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:50:35.718712 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.720112 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.721638 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.722380 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.722536 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:35.723141 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:50:35.724165 (+ 1024us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:50:35.724350 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:35.724453 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 16:50:35.724572 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":781,"thread_start_us":550,"threads_started":1}
I20260928 16:50:35.730176 17540 raft_consensus.cc:1261] T 80b1d8797951439ba8ddb8637cb65c27 P 3d8a14c88c364c6bb93a15a06febc220 [term 1 FOLLOWER]: Refusing update from remote peer 0aaabc4fbfc74e0297fcd463c14d495b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:50:35.730460 17680 raft_consensus.cc:1261] T 80b1d8797951439ba8ddb8637cb65c27 P b0f38fc994554643bf99c195029e1557 [term 1 FOLLOWER]: Refusing update from remote peer 0aaabc4fbfc74e0297fcd463c14d495b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:50:35.731180 17762 consensus_queue.cc:1035] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:50:35.731662 17805 consensus_queue.cc:1035] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:50:35.735759 17763 tablet_replica.cc:439] T 80b1d8797951439ba8ddb8637cb65c27 P 3d8a14c88c364c6bb93a15a06febc220: TxnStatusTablet state changed. Reason: New leader 0aaabc4fbfc74e0297fcd463c14d495b. Latest consensus state: current_term: 1 leader_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } } }
I20260928 16:50:35.736153 17763 tablet_replica.cc:442] T 80b1d8797951439ba8ddb8637cb65c27 P 3d8a14c88c364c6bb93a15a06febc220: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:50:35.738788 17800 mvcc.cc:205] Tried to move back new op lower bound from 7334355909541912576 to 7334355909360496640. Current Snapshot: MvccSnapshot[applied={T|T < 7334355909541912576}]
I20260928 16:50:35.741392 17798 mvcc.cc:205] Tried to move back new op lower bound from 7334355909541912576 to 7334355909360496640. Current Snapshot: MvccSnapshot[applied={T|T < 7334355909541912576}]
I20260928 16:50:35.742491 17763 tablet_replica.cc:439] T 80b1d8797951439ba8ddb8637cb65c27 P 3d8a14c88c364c6bb93a15a06febc220: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } } }
I20260928 16:50:35.742805 17763 tablet_replica.cc:442] T 80b1d8797951439ba8ddb8637cb65c27 P 3d8a14c88c364c6bb93a15a06febc220: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:50:35.747282 17775 tablet_replica.cc:439] T 80b1d8797951439ba8ddb8637cb65c27 P b0f38fc994554643bf99c195029e1557: TxnStatusTablet state changed. Reason: New leader 0aaabc4fbfc74e0297fcd463c14d495b. Latest consensus state: current_term: 1 leader_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } } }
I20260928 16:50:35.747700 17775 tablet_replica.cc:442] T 80b1d8797951439ba8ddb8637cb65c27 P b0f38fc994554643bf99c195029e1557: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:50:35.748114 17762 tablet_replica.cc:439] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } } }
I20260928 16:50:35.748473 17762 tablet_replica.cc:442] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:50:35.753019 17762 tablet_replica.cc:439] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } } }
I20260928 16:50:35.753386 17762 tablet_replica.cc:442] T 80b1d8797951439ba8ddb8637cb65c27 P 0aaabc4fbfc74e0297fcd463c14d495b: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:50:35.769204 17775 tablet_replica.cc:439] T 80b1d8797951439ba8ddb8637cb65c27 P b0f38fc994554643bf99c195029e1557: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } } }
I20260928 16:50:35.769575 17775 tablet_replica.cc:442] T 80b1d8797951439ba8ddb8637cb65c27 P b0f38fc994554643bf99c195029e1557: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:50:35.789273 17819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:35.782819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:40001 (local address 127.9.93.3:34421)
0928 16:50:35.783519 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:35.783538 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:35.783745 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:35.784194 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:35.784202 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:35.784217 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:50:35.784526 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:35.784537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.785796 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.785800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:35.786931 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:35.786947 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.787860 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.787866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:35.788057 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:35.788146 (+ 89us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:50:35.788572 (+ 426us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:50:35.788586 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 16:50:35.788672 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 16:50:35.788853 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":316,"threads_started":1}
I20260928 16:50:35.789240 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:35.782977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:34421 (local address 127.9.93.1:40001)
0928 16:50:35.783251 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:35.783259 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:35.783652 (+ 393us) server_negotiation.cc:407] Connection header received
0928 16:50:35.783927 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:35.783932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:35.784024 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:35.784132 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:50:35.784706 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.785638 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.787099 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.787648 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.787774 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:35.788343 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:50:35.788470 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:50:35.788558 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:35.788711 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 16:50:35.788814 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260928 16:50:35.839493 17804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:35.831475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:34511 (local address 127.9.93.3:37881)
0928 16:50:35.831679 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:35.831688 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:35.832242 (+ 554us) server_negotiation.cc:407] Connection header received
0928 16:50:35.832478 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:35.832485 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:35.832567 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:35.832651 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:50:35.833443 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.834681 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.836908 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.837717 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.837872 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:35.838519 (+ 647us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:50:35.838693 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:50:35.838790 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:35.838997 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 16:50:35.839110 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 16:50:35.839630 17821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:35.831410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:37881 (local address 127.9.93.1:34511)
0928 16:50:35.832116 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:35.832138 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:35.832329 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:35.832702 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:35.832707 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:35.832726 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:50:35.833217 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:35.833235 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.834829 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.834834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:35.836708 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:35.836729 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.837979 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.837987 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:35.838186 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:35.838288 (+ 102us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:50:35.838849 (+ 561us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:50:35.838860 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 16:50:35.838945 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 16:50:35.839145 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":343,"threads_started":1}
I20260928 16:50:35.886987 17239 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:50:35.887388 17239 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:50:35.891219 17239 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822: Sending DeleteTablet for 3 replicas of tablet f369a491339b4b4c85ef4111f7210290
I20260928 16:50:35.892270 17378 tablet_service.cc:1514] Processing DeleteTablet for tablet f369a491339b4b4c85ef4111f7210290 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:35 UTC) from {username='oryx', principal='oryx/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:33490
I20260928 16:50:35.892385 17519 tablet_service.cc:1514] Processing DeleteTablet for tablet f369a491339b4b4c85ef4111f7210290 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:35 UTC) from {username='oryx', principal='oryx/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:35056
I20260928 16:50:35.892963 17660 tablet_service.cc:1514] Processing DeleteTablet for tablet f369a491339b4b4c85ef4111f7210290 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:35 UTC) from {username='oryx', principal='oryx/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:37130
I20260928 16:50:35.893208 17824 tablet_replica.cc:330] T f369a491339b4b4c85ef4111f7210290 P 3d8a14c88c364c6bb93a15a06febc220: stopping tablet replica
I20260928 16:50:35.893503 17824 raft_consensus.cc:2229] T f369a491339b4b4c85ef4111f7210290 P 3d8a14c88c364c6bb93a15a06febc220 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:50:35.893982 17824 raft_consensus.cc:2258] T f369a491339b4b4c85ef4111f7210290 P 3d8a14c88c364c6bb93a15a06febc220 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:35.897769 17824 ts_tablet_manager.cc:1905] T f369a491339b4b4c85ef4111f7210290 P 3d8a14c88c364c6bb93a15a06febc220: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:50:35.901542 17825 tablet_replica.cc:330] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557: stopping tablet replica
I20260928 16:50:35.901845 17825 raft_consensus.cc:2229] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:50:35.902357 17825 raft_consensus.cc:2258] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:35.903861 17825 ts_tablet_manager.cc:1905] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:50:35.912510 17825 ts_tablet_manager.cc:1918] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:50:35.912709 17825 log.cc:1196] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/f369a491339b4b4c85ef4111f7210290
I20260928 16:50:35.913211 17825 ts_tablet_manager.cc:1939] T f369a491339b4b4c85ef4111f7210290 P b0f38fc994554643bf99c195029e1557: Deleting consensus metadata
I20260928 16:50:35.913287 17824 ts_tablet_manager.cc:1918] T f369a491339b4b4c85ef4111f7210290 P 3d8a14c88c364c6bb93a15a06febc220: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:50:35.913478 17824 log.cc:1196] T f369a491339b4b4c85ef4111f7210290 P 3d8a14c88c364c6bb93a15a06febc220: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/f369a491339b4b4c85ef4111f7210290
I20260928 16:50:35.913996 17824 ts_tablet_manager.cc:1939] T f369a491339b4b4c85ef4111f7210290 P 3d8a14c88c364c6bb93a15a06febc220: Deleting consensus metadata
I20260928 16:50:35.914714 17224 catalog_manager.cc:4841] TS b0f38fc994554643bf99c195029e1557 (127.9.93.3:37881): tablet f369a491339b4b4c85ef4111f7210290 (table test-table [id=8876c80efb764023b0314c6b02242e8b]) successfully deleted
I20260928 16:50:35.915400 17224 catalog_manager.cc:4841] TS 3d8a14c88c364c6bb93a15a06febc220 (127.9.93.2:33729): tablet f369a491339b4b4c85ef4111f7210290 (table test-table [id=8876c80efb764023b0314c6b02242e8b]) successfully deleted
I20260928 16:50:35.915542 17823 tablet_replica.cc:330] T f369a491339b4b4c85ef4111f7210290 P 0aaabc4fbfc74e0297fcd463c14d495b: stopping tablet replica
I20260928 16:50:35.915832 17823 raft_consensus.cc:2229] T f369a491339b4b4c85ef4111f7210290 P 0aaabc4fbfc74e0297fcd463c14d495b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:50:35.916175 17823 raft_consensus.cc:2258] T f369a491339b4b4c85ef4111f7210290 P 0aaabc4fbfc74e0297fcd463c14d495b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:35.917476 17823 ts_tablet_manager.cc:1905] T f369a491339b4b4c85ef4111f7210290 P 0aaabc4fbfc74e0297fcd463c14d495b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:50:35.926425 17823 ts_tablet_manager.cc:1918] T f369a491339b4b4c85ef4111f7210290 P 0aaabc4fbfc74e0297fcd463c14d495b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:50:35.926631 17823 log.cc:1196] T f369a491339b4b4c85ef4111f7210290 P 0aaabc4fbfc74e0297fcd463c14d495b: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-0/wal/wals/f369a491339b4b4c85ef4111f7210290
I20260928 16:50:35.927243 17823 ts_tablet_manager.cc:1939] T f369a491339b4b4c85ef4111f7210290 P 0aaabc4fbfc74e0297fcd463c14d495b: Deleting consensus metadata
I20260928 16:50:35.928646 17224 catalog_manager.cc:4841] TS 0aaabc4fbfc74e0297fcd463c14d495b (127.9.93.1:40001): tablet f369a491339b4b4c85ef4111f7210290 (table test-table [id=8876c80efb764023b0314c6b02242e8b]) successfully deleted
Sep 28 16:50:35 dist-test-slave-wzcb krb5kdc[17194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 16:50:35 dist-test-slave-wzcb krb5kdc[17194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:35 dist-test-slave-wzcb krb5kdc[17194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 16:50:35.989753 17834 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:35.957645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39880 (local address 127.9.93.62:33613)
0928 16:50:35.976423 (+ 18778us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:35.976428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:35.976447 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:50:35.976506 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:35.976511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:35.976576 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:35.976690 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:35.978531 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.979506 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.980950 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.981420 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:35.981561 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:35.989407 (+ 7846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39880: BlockingRecv error: recv got EOF from 127.0.0.1:39880 (error 108)
Metrics: {"server-negotiator.queue_time_us":18646,"thread_start_us":18475,"threads_started":1}
Sep 28 16:50:36 dist-test-slave-wzcb krb5kdc[17194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM
I20260928 16:50:36.013770 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:35.995595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39892 (local address 127.9.93.62:33613)
0928 16:50:35.995824 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:35.995831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:35.996214 (+ 383us) server_negotiation.cc:407] Connection header received
0928 16:50:35.996489 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:35.996493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:35.996578 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:35.996672 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:35.998526 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:35.999393 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:36.001374 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:36.001848 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:36.001994 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:36.007464 (+ 5470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:36.007471 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:36.007474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:36.007523 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:36.010456 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:36.011547 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:36.011553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:36.011556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:36.011763 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:36.012223 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:36.012229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:36.012231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:36.012593 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:36.012810 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:36.013281 (+ 471us) server_negotiation.cc:299] Negotiation successful
0928 16:50:36.013494 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 16:50:36.017946 17239 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:50:36.018653 17239 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:50:36.026888 17660 tablet_service.cc:1467] Processing CreateTablet for tablet 15edb3d9bcd44b3b865f07f4b630b362 (DEFAULT_TABLE table=test-table [id=0a11bad0dadc4dc58ba9fe5c2b5fec35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:50:36.026911 17519 tablet_service.cc:1467] Processing CreateTablet for tablet 15edb3d9bcd44b3b865f07f4b630b362 (DEFAULT_TABLE table=test-table [id=0a11bad0dadc4dc58ba9fe5c2b5fec35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:50:36.026903 17378 tablet_service.cc:1467] Processing CreateTablet for tablet 15edb3d9bcd44b3b865f07f4b630b362 (DEFAULT_TABLE table=test-table [id=0a11bad0dadc4dc58ba9fe5c2b5fec35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:50:36.027513 17660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15edb3d9bcd44b3b865f07f4b630b362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:36.027514 17519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15edb3d9bcd44b3b865f07f4b630b362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:36.027576 17378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15edb3d9bcd44b3b865f07f4b630b362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:36.041923 17842 tablet_bootstrap.cc:492] T 15edb3d9bcd44b3b865f07f4b630b362 P b0f38fc994554643bf99c195029e1557: Bootstrap starting.
I20260928 16:50:36.042824 17840 tablet_bootstrap.cc:492] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b: Bootstrap starting.
I20260928 16:50:36.042928 17841 tablet_bootstrap.cc:492] T 15edb3d9bcd44b3b865f07f4b630b362 P 3d8a14c88c364c6bb93a15a06febc220: Bootstrap starting.
I20260928 16:50:36.044314 17842 tablet_bootstrap.cc:654] T 15edb3d9bcd44b3b865f07f4b630b362 P b0f38fc994554643bf99c195029e1557: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:36.046614 17841 tablet_bootstrap.cc:654] T 15edb3d9bcd44b3b865f07f4b630b362 P 3d8a14c88c364c6bb93a15a06febc220: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:36.047257 17840 tablet_bootstrap.cc:654] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:36.048106 17842 tablet_bootstrap.cc:492] T 15edb3d9bcd44b3b865f07f4b630b362 P b0f38fc994554643bf99c195029e1557: No bootstrap required, opened a new log
I20260928 16:50:36.048314 17842 ts_tablet_manager.cc:1397] T 15edb3d9bcd44b3b865f07f4b630b362 P b0f38fc994554643bf99c195029e1557: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 16:50:36.049305 17842 raft_consensus.cc:348] T 15edb3d9bcd44b3b865f07f4b630b362 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:36.049643 17842 raft_consensus.cc:374] T 15edb3d9bcd44b3b865f07f4b630b362 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:36.049762 17842 raft_consensus.cc:729] T 15edb3d9bcd44b3b865f07f4b630b362 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f38fc994554643bf99c195029e1557, State: Initialized, Role: FOLLOWER
I20260928 16:50:36.050166 17842 consensus_queue.cc:260] T 15edb3d9bcd44b3b865f07f4b630b362 P b0f38fc994554643bf99c195029e1557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:36.054467 17842 ts_tablet_manager.cc:1428] T 15edb3d9bcd44b3b865f07f4b630b362 P b0f38fc994554643bf99c195029e1557: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260928 16:50:36.059269 17841 tablet_bootstrap.cc:492] T 15edb3d9bcd44b3b865f07f4b630b362 P 3d8a14c88c364c6bb93a15a06febc220: No bootstrap required, opened a new log
I20260928 16:50:36.059556 17841 ts_tablet_manager.cc:1397] T 15edb3d9bcd44b3b865f07f4b630b362 P 3d8a14c88c364c6bb93a15a06febc220: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.004s
I20260928 16:50:36.060582 17841 raft_consensus.cc:348] T 15edb3d9bcd44b3b865f07f4b630b362 P 3d8a14c88c364c6bb93a15a06febc220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:36.060873 17841 raft_consensus.cc:374] T 15edb3d9bcd44b3b865f07f4b630b362 P 3d8a14c88c364c6bb93a15a06febc220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:36.060988 17841 raft_consensus.cc:729] T 15edb3d9bcd44b3b865f07f4b630b362 P 3d8a14c88c364c6bb93a15a06febc220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d8a14c88c364c6bb93a15a06febc220, State: Initialized, Role: FOLLOWER
I20260928 16:50:36.061095 17840 tablet_bootstrap.cc:492] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b: No bootstrap required, opened a new log
I20260928 16:50:36.061311 17840 ts_tablet_manager.cc:1397] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.000s
I20260928 16:50:36.061348 17841 consensus_queue.cc:260] T 15edb3d9bcd44b3b865f07f4b630b362 P 3d8a14c88c364c6bb93a15a06febc220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:36.062188 17841 ts_tablet_manager.cc:1428] T 15edb3d9bcd44b3b865f07f4b630b362 P 3d8a14c88c364c6bb93a15a06febc220: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 16:50:36.062376 17840 raft_consensus.cc:348] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:36.062784 17840 raft_consensus.cc:374] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:36.062907 17840 raft_consensus.cc:729] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aaabc4fbfc74e0297fcd463c14d495b, State: Initialized, Role: FOLLOWER
I20260928 16:50:36.063271 17840 consensus_queue.cc:260] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:36.064253 17840 ts_tablet_manager.cc:1428] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 16:50:36.120927 17805 raft_consensus.cc:482] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:50:36.121263 17805 raft_consensus.cc:504] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:36.122071 17805 leader_election.cc:288] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d8a14c88c364c6bb93a15a06febc220 (127.9.93.2:33729), b0f38fc994554643bf99c195029e1557 (127.9.93.3:37881)
I20260928 16:50:36.122884 17540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15edb3d9bcd44b3b865f07f4b630b362" candidate_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d8a14c88c364c6bb93a15a06febc220" is_pre_election: true
I20260928 16:50:36.123077 17680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15edb3d9bcd44b3b865f07f4b630b362" candidate_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f38fc994554643bf99c195029e1557" is_pre_election: true
I20260928 16:50:36.123281 17540 raft_consensus.cc:2454] T 15edb3d9bcd44b3b865f07f4b630b362 P 3d8a14c88c364c6bb93a15a06febc220 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aaabc4fbfc74e0297fcd463c14d495b in term 0.
I20260928 16:50:36.123409 17680 raft_consensus.cc:2454] T 15edb3d9bcd44b3b865f07f4b630b362 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aaabc4fbfc74e0297fcd463c14d495b in term 0.
I20260928 16:50:36.124003 17329 leader_election.cc:302] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aaabc4fbfc74e0297fcd463c14d495b, b0f38fc994554643bf99c195029e1557; no voters:
I20260928 16:50:36.124389 17805 raft_consensus.cc:2781] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:50:36.124518 17805 raft_consensus.cc:482] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:50:36.124646 17805 raft_consensus.cc:3037] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:36.128528 17805 raft_consensus.cc:504] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:36.129525 17805 leader_election.cc:288] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b [CANDIDATE]: Term 1 election: Requested vote from peers 3d8a14c88c364c6bb93a15a06febc220 (127.9.93.2:33729), b0f38fc994554643bf99c195029e1557 (127.9.93.3:37881)
I20260928 16:50:36.130300 17680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15edb3d9bcd44b3b865f07f4b630b362" candidate_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f38fc994554643bf99c195029e1557"
I20260928 16:50:36.130287 17540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15edb3d9bcd44b3b865f07f4b630b362" candidate_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d8a14c88c364c6bb93a15a06febc220"
I20260928 16:50:36.130568 17680 raft_consensus.cc:3037] T 15edb3d9bcd44b3b865f07f4b630b362 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:36.130611 17540 raft_consensus.cc:3037] T 15edb3d9bcd44b3b865f07f4b630b362 P 3d8a14c88c364c6bb93a15a06febc220 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:36.133131 17680 raft_consensus.cc:2454] T 15edb3d9bcd44b3b865f07f4b630b362 P b0f38fc994554643bf99c195029e1557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aaabc4fbfc74e0297fcd463c14d495b in term 1.
I20260928 16:50:36.133773 17329 leader_election.cc:302] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aaabc4fbfc74e0297fcd463c14d495b, b0f38fc994554643bf99c195029e1557; no voters:
I20260928 16:50:36.134179 17805 raft_consensus.cc:2781] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:50:36.134145 17540 raft_consensus.cc:2454] T 15edb3d9bcd44b3b865f07f4b630b362 P 3d8a14c88c364c6bb93a15a06febc220 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aaabc4fbfc74e0297fcd463c14d495b in term 1.
I20260928 16:50:36.134501 17805 raft_consensus.cc:686] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b [term 1 LEADER]: Becoming Leader. State: Replica: 0aaabc4fbfc74e0297fcd463c14d495b, State: Running, Role: LEADER
I20260928 16:50:36.134866 17805 consensus_queue.cc:237] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:36.138465 17239 catalog_manager.cc:5495] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0aaabc4fbfc74e0297fcd463c14d495b (127.9.93.1). New cstate: current_term: 1 leader_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } health_report { overall_health: UNKNOWN } } }
I20260928 16:50:36.156262 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:36.150136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33512 (local address 127.9.93.1:40001)
0928 16:50:36.150361 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:36.150368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:36.150398 (+ 30us) server_negotiation.cc:407] Connection header received
0928 16:50:36.150520 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:36.150526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:36.150657 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:36.150772 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:50:36.151118 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:36.152414 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:36.153875 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:36.154570 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:36.154703 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:36.155190 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:50:36.155405 (+ 215us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:50:36.155493 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:36.155787 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 16:50:36.155891 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:50:36.160708 17680 raft_consensus.cc:1261] T 15edb3d9bcd44b3b865f07f4b630b362 P b0f38fc994554643bf99c195029e1557 [term 1 FOLLOWER]: Refusing update from remote peer 0aaabc4fbfc74e0297fcd463c14d495b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:50:36.160888 17540 raft_consensus.cc:1261] T 15edb3d9bcd44b3b865f07f4b630b362 P 3d8a14c88c364c6bb93a15a06febc220 [term 1 FOLLOWER]: Refusing update from remote peer 0aaabc4fbfc74e0297fcd463c14d495b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:50:36.161598 17762 consensus_queue.cc:1035] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:50:36.161978 17805 consensus_queue.cc:1035] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:50:36.181360 17238 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:50:36.181746 17238 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:50:36.186309 17238 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822: Sending DeleteTablet for 3 replicas of tablet 15edb3d9bcd44b3b865f07f4b630b362
I20260928 16:50:36.187517 17660 tablet_service.cc:1514] Processing DeleteTablet for tablet 15edb3d9bcd44b3b865f07f4b630b362 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:36 UTC) from {username='oryx', principal='oryx/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:37130
I20260928 16:50:36.187759 17378 tablet_service.cc:1514] Processing DeleteTablet for tablet 15edb3d9bcd44b3b865f07f4b630b362 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:36 UTC) from {username='oryx', principal='oryx/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:33490
I20260928 16:50:36.187884 17825 tablet_replica.cc:330] T 15edb3d9bcd44b3b865f07f4b630b362 P b0f38fc994554643bf99c195029e1557: stopping tablet replica
I20260928 16:50:36.188118 17825 raft_consensus.cc:2229] T 15edb3d9bcd44b3b865f07f4b630b362 P b0f38fc994554643bf99c195029e1557 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:50:36.188150 17377 tablet_service.cc:1514] Processing DeleteTablet for tablet 15edb3d9bcd44b3b865f07f4b630b362 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:36 UTC) from {username='oryx', principal='oryx/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:33490
I20260928 16:50:36.188158 17823 tablet_replica.cc:330] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b: stopping tablet replica
I20260928 16:50:36.188387 17825 raft_consensus.cc:2258] T 15edb3d9bcd44b3b865f07f4b630b362 P b0f38fc994554643bf99c195029e1557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:36.188575 17823 raft_consensus.cc:2229] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:50:36.189252 17823 raft_consensus.cc:2258] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:36.189260 17660 tablet_service.cc:1514] Processing DeleteTablet for tablet 15edb3d9bcd44b3b865f07f4b630b362 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:36 UTC) from {username='oryx', principal='oryx/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:37130
I20260928 16:50:36.189700 17519 tablet_service.cc:1514] Processing DeleteTablet for tablet 15edb3d9bcd44b3b865f07f4b630b362 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:36 UTC) from {username='oryx', principal='oryx/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:35056
I20260928 16:50:36.190068 17824 tablet_replica.cc:330] T 15edb3d9bcd44b3b865f07f4b630b362 P 3d8a14c88c364c6bb93a15a06febc220: stopping tablet replica
I20260928 16:50:36.190317 17824 raft_consensus.cc:2229] T 15edb3d9bcd44b3b865f07f4b630b362 P 3d8a14c88c364c6bb93a15a06febc220 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:50:36.190454 17823 ts_tablet_manager.cc:1905] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:50:36.190608 17824 raft_consensus.cc:2258] T 15edb3d9bcd44b3b865f07f4b630b362 P 3d8a14c88c364c6bb93a15a06febc220 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:36.191414 17239 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 16:50:36.191987 17825 ts_tablet_manager.cc:1905] T 15edb3d9bcd44b3b865f07f4b630b362 P b0f38fc994554643bf99c195029e1557: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:50:36.192389 17824 ts_tablet_manager.cc:1905] T 15edb3d9bcd44b3b865f07f4b630b362 P 3d8a14c88c364c6bb93a15a06febc220: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 16:50:36.192620 17239 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:50:36.197963 17823 ts_tablet_manager.cc:1918] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:50:36.198170 17823 log.cc:1196] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-0/wal/wals/15edb3d9bcd44b3b865f07f4b630b362
I20260928 16:50:36.198683 17823 ts_tablet_manager.cc:1939] T 15edb3d9bcd44b3b865f07f4b630b362 P 0aaabc4fbfc74e0297fcd463c14d495b: Deleting consensus metadata
I20260928 16:50:36.199638 17825 ts_tablet_manager.cc:1918] T 15edb3d9bcd44b3b865f07f4b630b362 P b0f38fc994554643bf99c195029e1557: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:50:36.199846 17825 log.cc:1196] T 15edb3d9bcd44b3b865f07f4b630b362 P b0f38fc994554643bf99c195029e1557: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/15edb3d9bcd44b3b865f07f4b630b362
I20260928 16:50:36.199867 17824 ts_tablet_manager.cc:1918] T 15edb3d9bcd44b3b865f07f4b630b362 P 3d8a14c88c364c6bb93a15a06febc220: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:50:36.200049 17824 log.cc:1196] T 15edb3d9bcd44b3b865f07f4b630b362 P 3d8a14c88c364c6bb93a15a06febc220: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/15edb3d9bcd44b3b865f07f4b630b362
I20260928 16:50:36.200311 17825 ts_tablet_manager.cc:1939] T 15edb3d9bcd44b3b865f07f4b630b362 P b0f38fc994554643bf99c195029e1557: Deleting consensus metadata
I20260928 16:50:36.200438 17824 ts_tablet_manager.cc:1939] T 15edb3d9bcd44b3b865f07f4b630b362 P 3d8a14c88c364c6bb93a15a06febc220: Deleting consensus metadata
I20260928 16:50:36.202384 17224 catalog_manager.cc:4841] TS 3d8a14c88c364c6bb93a15a06febc220 (127.9.93.2:33729): tablet 15edb3d9bcd44b3b865f07f4b630b362 (table test-table [id=0a11bad0dadc4dc58ba9fe5c2b5fec35]) successfully deleted
I20260928 16:50:36.202674 17224 catalog_manager.cc:4841] TS b0f38fc994554643bf99c195029e1557 (127.9.93.3:37881): tablet 15edb3d9bcd44b3b865f07f4b630b362 (table test-table [id=0a11bad0dadc4dc58ba9fe5c2b5fec35]) successfully deleted
I20260928 16:50:36.202991 17660 tablet_service.cc:1467] Processing CreateTablet for tablet 2e3fc13bfb824e97b136bf1253ec0b36 (DEFAULT_TABLE table=test-table [id=d4e186a4f6fa470db14adf7c794a066e]), partition=RANGE (key) PARTITION UNBOUNDED
W20260928 16:50:36.203088 17224 catalog_manager.cc:4808] TS b0f38fc994554643bf99c195029e1557 (127.9.93.3:37881): delete failed for tablet 15edb3d9bcd44b3b865f07f4b630b362 because the tablet was not found. No further retry: Not found: Tablet not found: 15edb3d9bcd44b3b865f07f4b630b362
I20260928 16:50:36.203509 17660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e3fc13bfb824e97b136bf1253ec0b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:36.204053 17377 tablet_service.cc:1467] Processing CreateTablet for tablet 2e3fc13bfb824e97b136bf1253ec0b36 (DEFAULT_TABLE table=test-table [id=d4e186a4f6fa470db14adf7c794a066e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:50:36.204608 17377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e3fc13bfb824e97b136bf1253ec0b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:36.210356 17840 tablet_bootstrap.cc:492] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b: Bootstrap starting.
I20260928 16:50:36.211042 17519 tablet_service.cc:1467] Processing CreateTablet for tablet 2e3fc13bfb824e97b136bf1253ec0b36 (DEFAULT_TABLE table=test-table [id=d4e186a4f6fa470db14adf7c794a066e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:50:36.211623 17842 tablet_bootstrap.cc:492] T 2e3fc13bfb824e97b136bf1253ec0b36 P b0f38fc994554643bf99c195029e1557: Bootstrap starting.
I20260928 16:50:36.211697 17519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e3fc13bfb824e97b136bf1253ec0b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:36.212453 17840 tablet_bootstrap.cc:654] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:36.213476 17842 tablet_bootstrap.cc:654] T 2e3fc13bfb824e97b136bf1253ec0b36 P b0f38fc994554643bf99c195029e1557: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:36.216413 17224 catalog_manager.cc:4841] TS 0aaabc4fbfc74e0297fcd463c14d495b (127.9.93.1:40001): tablet 15edb3d9bcd44b3b865f07f4b630b362 (table test-table [id=0a11bad0dadc4dc58ba9fe5c2b5fec35]) successfully deleted
W20260928 16:50:36.216873 17224 catalog_manager.cc:4808] TS 0aaabc4fbfc74e0297fcd463c14d495b (127.9.93.1:40001): delete failed for tablet 15edb3d9bcd44b3b865f07f4b630b362 because the tablet was not found. No further retry: Not found: Tablet not found: 15edb3d9bcd44b3b865f07f4b630b362
I20260928 16:50:36.221992 17841 tablet_bootstrap.cc:492] T 2e3fc13bfb824e97b136bf1253ec0b36 P 3d8a14c88c364c6bb93a15a06febc220: Bootstrap starting.
I20260928 16:50:36.222784 17842 tablet_bootstrap.cc:492] T 2e3fc13bfb824e97b136bf1253ec0b36 P b0f38fc994554643bf99c195029e1557: No bootstrap required, opened a new log
I20260928 16:50:36.223023 17842 ts_tablet_manager.cc:1397] T 2e3fc13bfb824e97b136bf1253ec0b36 P b0f38fc994554643bf99c195029e1557: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.007s
I20260928 16:50:36.224326 17841 tablet_bootstrap.cc:654] T 2e3fc13bfb824e97b136bf1253ec0b36 P 3d8a14c88c364c6bb93a15a06febc220: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:36.224546 17842 raft_consensus.cc:348] T 2e3fc13bfb824e97b136bf1253ec0b36 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:36.224849 17842 raft_consensus.cc:374] T 2e3fc13bfb824e97b136bf1253ec0b36 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:36.224925 17842 raft_consensus.cc:729] T 2e3fc13bfb824e97b136bf1253ec0b36 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f38fc994554643bf99c195029e1557, State: Initialized, Role: FOLLOWER
I20260928 16:50:36.225242 17842 consensus_queue.cc:260] T 2e3fc13bfb824e97b136bf1253ec0b36 P b0f38fc994554643bf99c195029e1557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:36.226389 17842 ts_tablet_manager.cc:1428] T 2e3fc13bfb824e97b136bf1253ec0b36 P b0f38fc994554643bf99c195029e1557: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 16:50:36.237272 17840 tablet_bootstrap.cc:492] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b: No bootstrap required, opened a new log
I20260928 16:50:36.237269 17841 tablet_bootstrap.cc:492] T 2e3fc13bfb824e97b136bf1253ec0b36 P 3d8a14c88c364c6bb93a15a06febc220: No bootstrap required, opened a new log
I20260928 16:50:36.237527 17840 ts_tablet_manager.cc:1397] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.017s
I20260928 16:50:36.237689 17841 ts_tablet_manager.cc:1397] T 2e3fc13bfb824e97b136bf1253ec0b36 P 3d8a14c88c364c6bb93a15a06febc220: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20260928 16:50:36.238617 17841 raft_consensus.cc:348] T 2e3fc13bfb824e97b136bf1253ec0b36 P 3d8a14c88c364c6bb93a15a06febc220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:36.238593 17840 raft_consensus.cc:348] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:36.238941 17840 raft_consensus.cc:374] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:36.239039 17840 raft_consensus.cc:729] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aaabc4fbfc74e0297fcd463c14d495b, State: Initialized, Role: FOLLOWER
I20260928 16:50:36.239140 17841 raft_consensus.cc:374] T 2e3fc13bfb824e97b136bf1253ec0b36 P 3d8a14c88c364c6bb93a15a06febc220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:36.239328 17841 raft_consensus.cc:729] T 2e3fc13bfb824e97b136bf1253ec0b36 P 3d8a14c88c364c6bb93a15a06febc220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d8a14c88c364c6bb93a15a06febc220, State: Initialized, Role: FOLLOWER
I20260928 16:50:36.239390 17840 consensus_queue.cc:260] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:36.239615 17841 consensus_queue.cc:260] T 2e3fc13bfb824e97b136bf1253ec0b36 P 3d8a14c88c364c6bb93a15a06febc220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:36.240374 17841 ts_tablet_manager.cc:1428] T 2e3fc13bfb824e97b136bf1253ec0b36 P 3d8a14c88c364c6bb93a15a06febc220: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 16:50:36.240909 17840 ts_tablet_manager.cc:1428] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 16:50:36.257565 17815 raft_consensus.cc:482] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:50:36.257812 17815 raft_consensus.cc:504] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:36.258651 17815 leader_election.cc:288] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d8a14c88c364c6bb93a15a06febc220 (127.9.93.2:33729), b0f38fc994554643bf99c195029e1557 (127.9.93.3:37881)
I20260928 16:50:36.259413 17540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e3fc13bfb824e97b136bf1253ec0b36" candidate_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d8a14c88c364c6bb93a15a06febc220" is_pre_election: true
I20260928 16:50:36.259752 17540 raft_consensus.cc:2454] T 2e3fc13bfb824e97b136bf1253ec0b36 P 3d8a14c88c364c6bb93a15a06febc220 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aaabc4fbfc74e0297fcd463c14d495b in term 0.
I20260928 16:50:36.260241 17329 leader_election.cc:302] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aaabc4fbfc74e0297fcd463c14d495b, 3d8a14c88c364c6bb93a15a06febc220; no voters:
I20260928 16:50:36.260484 17679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e3fc13bfb824e97b136bf1253ec0b36" candidate_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f38fc994554643bf99c195029e1557" is_pre_election: true
I20260928 16:50:36.260864 17679 raft_consensus.cc:2454] T 2e3fc13bfb824e97b136bf1253ec0b36 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aaabc4fbfc74e0297fcd463c14d495b in term 0.
I20260928 16:50:36.260952 17762 raft_consensus.cc:2781] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:50:36.261099 17762 raft_consensus.cc:482] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:50:36.261224 17762 raft_consensus.cc:3037] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:36.264386 17762 raft_consensus.cc:504] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:36.265097 17762 leader_election.cc:288] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b [CANDIDATE]: Term 1 election: Requested vote from peers 3d8a14c88c364c6bb93a15a06febc220 (127.9.93.2:33729), b0f38fc994554643bf99c195029e1557 (127.9.93.3:37881)
I20260928 16:50:36.265715 17540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e3fc13bfb824e97b136bf1253ec0b36" candidate_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d8a14c88c364c6bb93a15a06febc220"
I20260928 16:50:36.265818 17679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e3fc13bfb824e97b136bf1253ec0b36" candidate_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f38fc994554643bf99c195029e1557"
I20260928 16:50:36.265991 17540 raft_consensus.cc:3037] T 2e3fc13bfb824e97b136bf1253ec0b36 P 3d8a14c88c364c6bb93a15a06febc220 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:36.266106 17679 raft_consensus.cc:3037] T 2e3fc13bfb824e97b136bf1253ec0b36 P b0f38fc994554643bf99c195029e1557 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:36.268589 17540 raft_consensus.cc:2454] T 2e3fc13bfb824e97b136bf1253ec0b36 P 3d8a14c88c364c6bb93a15a06febc220 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aaabc4fbfc74e0297fcd463c14d495b in term 1.
I20260928 16:50:36.269166 17329 leader_election.cc:302] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aaabc4fbfc74e0297fcd463c14d495b, 3d8a14c88c364c6bb93a15a06febc220; no voters:
I20260928 16:50:36.269526 17762 raft_consensus.cc:2781] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:50:36.269688 17762 raft_consensus.cc:686] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b [term 1 LEADER]: Becoming Leader. State: Replica: 0aaabc4fbfc74e0297fcd463c14d495b, State: Running, Role: LEADER
I20260928 16:50:36.269681 17679 raft_consensus.cc:2454] T 2e3fc13bfb824e97b136bf1253ec0b36 P b0f38fc994554643bf99c195029e1557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aaabc4fbfc74e0297fcd463c14d495b in term 1.
I20260928 16:50:36.270053 17762 consensus_queue.cc:237] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } }
I20260928 16:50:36.273365 17239 catalog_manager.cc:5495] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0aaabc4fbfc74e0297fcd463c14d495b (127.9.93.1). New cstate: current_term: 1 leader_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaabc4fbfc74e0297fcd463c14d495b" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 40001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 } health_report { overall_health: UNKNOWN } } }
I20260928 16:50:36.314079 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:36.308017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:57441 (local address 127.9.93.1:40001)
0928 16:50:36.308169 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:36.308174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:36.308193 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:50:36.308287 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:36.308294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:36.308361 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:36.308440 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:50:36.308923 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:36.309708 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:36.311998 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:36.312539 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:36.312676 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:36.313240 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:50:36.313387 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:50:36.313469 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:36.313661 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 16:50:36.313745 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 16:50:36.314185 17821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:36.307742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:40001 (local address 127.9.93.1:57441)
0928 16:50:36.308007 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:36.308030 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:36.308175 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:36.308489 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:36.308495 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:36.308508 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:50:36.308741 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:36.308753 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:36.309903 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:36.309909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:36.311792 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:36.311819 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:36.312756 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:36.312762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:36.312971 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:36.313031 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:50:36.313504 (+ 473us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:50:36.313519 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 16:50:36.313619 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 16:50:36.313816 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 16:50:36.324637 17679 raft_consensus.cc:1261] T 2e3fc13bfb824e97b136bf1253ec0b36 P b0f38fc994554643bf99c195029e1557 [term 1 FOLLOWER]: Refusing update from remote peer 0aaabc4fbfc74e0297fcd463c14d495b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:50:36.324899 17540 raft_consensus.cc:1261] T 2e3fc13bfb824e97b136bf1253ec0b36 P 3d8a14c88c364c6bb93a15a06febc220 [term 1 FOLLOWER]: Refusing update from remote peer 0aaabc4fbfc74e0297fcd463c14d495b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:50:36.325320 17805 consensus_queue.cc:1035] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0f38fc994554643bf99c195029e1557" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:50:36.325659 17855 consensus_queue.cc:1035] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8a14c88c364c6bb93a15a06febc220" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:50:36.409870 17238 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:50:36.410192 17238 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:50:36.413784 17238 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58662c670a204a0d9fbf0403a4ea1822: Sending DeleteTablet for 3 replicas of tablet 2e3fc13bfb824e97b136bf1253ec0b36
I20260928 16:50:36.414867 17378 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e3fc13bfb824e97b136bf1253ec0b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:36 UTC) from {username='oryx', principal='oryx/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:33490
I20260928 16:50:36.415289 17519 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e3fc13bfb824e97b136bf1253ec0b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:36 UTC) from {username='oryx', principal='oryx/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:35056
I20260928 16:50:36.415637 17660 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e3fc13bfb824e97b136bf1253ec0b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:50:36 UTC) from {username='oryx', principal='oryx/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:37130
I20260928 16:50:36.415771 17823 tablet_replica.cc:330] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b: stopping tablet replica
I20260928 16:50:36.416014 17825 tablet_replica.cc:330] T 2e3fc13bfb824e97b136bf1253ec0b36 P b0f38fc994554643bf99c195029e1557: stopping tablet replica
I20260928 16:50:36.416023 17823 raft_consensus.cc:2229] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:50:36.416251 17825 raft_consensus.cc:2229] T 2e3fc13bfb824e97b136bf1253ec0b36 P b0f38fc994554643bf99c195029e1557 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:50:36.416488 17825 raft_consensus.cc:2258] T 2e3fc13bfb824e97b136bf1253ec0b36 P b0f38fc994554643bf99c195029e1557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:36.416492 17823 raft_consensus.cc:2258] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:36.416904 17824 tablet_replica.cc:330] T 2e3fc13bfb824e97b136bf1253ec0b36 P 3d8a14c88c364c6bb93a15a06febc220: stopping tablet replica
I20260928 16:50:36.417116 17824 raft_consensus.cc:2229] T 2e3fc13bfb824e97b136bf1253ec0b36 P 3d8a14c88c364c6bb93a15a06febc220 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:50:36.417395 17824 raft_consensus.cc:2258] T 2e3fc13bfb824e97b136bf1253ec0b36 P 3d8a14c88c364c6bb93a15a06febc220 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:50:36.418112 17825 ts_tablet_manager.cc:1905] T 2e3fc13bfb824e97b136bf1253ec0b36 P b0f38fc994554643bf99c195029e1557: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:50:36.418783 17824 ts_tablet_manager.cc:1905] T 2e3fc13bfb824e97b136bf1253ec0b36 P 3d8a14c88c364c6bb93a15a06febc220: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:50:36.418946 17823 ts_tablet_manager.cc:1905] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:50:36.425984 17825 ts_tablet_manager.cc:1918] T 2e3fc13bfb824e97b136bf1253ec0b36 P b0f38fc994554643bf99c195029e1557: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:50:36.426185 17825 log.cc:1196] T 2e3fc13bfb824e97b136bf1253ec0b36 P b0f38fc994554643bf99c195029e1557: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/2e3fc13bfb824e97b136bf1253ec0b36
I20260928 16:50:36.426402 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 17312
I20260928 16:50:36.426659 17825 ts_tablet_manager.cc:1939] T 2e3fc13bfb824e97b136bf1253ec0b36 P b0f38fc994554643bf99c195029e1557: Deleting consensus metadata
I20260928 16:50:36.426928 17824 ts_tablet_manager.cc:1918] T 2e3fc13bfb824e97b136bf1253ec0b36 P 3d8a14c88c364c6bb93a15a06febc220: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:50:36.427098 17824 log.cc:1196] T 2e3fc13bfb824e97b136bf1253ec0b36 P 3d8a14c88c364c6bb93a15a06febc220: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/2e3fc13bfb824e97b136bf1253ec0b36
I20260928 16:50:36.427857 17824 ts_tablet_manager.cc:1939] T 2e3fc13bfb824e97b136bf1253ec0b36 P 3d8a14c88c364c6bb93a15a06febc220: Deleting consensus metadata
I20260928 16:50:36.428378 17224 catalog_manager.cc:4841] TS b0f38fc994554643bf99c195029e1557 (127.9.93.3:37881): tablet 2e3fc13bfb824e97b136bf1253ec0b36 (table test-table [id=d4e186a4f6fa470db14adf7c794a066e]) successfully deleted
I20260928 16:50:36.429279 17224 catalog_manager.cc:4841] TS 3d8a14c88c364c6bb93a15a06febc220 (127.9.93.2:33729): tablet 2e3fc13bfb824e97b136bf1253ec0b36 (table test-table [id=d4e186a4f6fa470db14adf7c794a066e]) successfully deleted
I20260928 16:50:36.430596 17823 ts_tablet_manager.cc:1918] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:50:36.430760 17823 log.cc:1196] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790614166338986-9588-0/minicluster-data/ts-0/wal/wals/2e3fc13bfb824e97b136bf1253ec0b36
I20260928 16:50:36.431165 17823 ts_tablet_manager.cc:1939] T 2e3fc13bfb824e97b136bf1253ec0b36 P 0aaabc4fbfc74e0297fcd463c14d495b: Deleting consensus metadata
I20260928 16:50:36.432456 17224 catalog_manager.cc:4841] TS 0aaabc4fbfc74e0297fcd463c14d495b (127.9.93.1:40001): tablet 2e3fc13bfb824e97b136bf1253ec0b36 (table test-table [id=d4e186a4f6fa470db14adf7c794a066e]) successfully deleted
Sep 28 16:50:36 dist-test-slave-wzcb krb5kdc[17194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.93.1@KRBTEST.COM
I20260928 16:50:36.449952 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:36.430088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33518 (local address 127.9.93.1:40001)
0928 16:50:36.430295 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:36.430301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:36.432687 (+ 2386us) server_negotiation.cc:407] Connection header received
0928 16:50:36.432983 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:36.432988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:36.433073 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:36.433188 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:36.434518 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:36.435423 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:36.436809 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:36.437327 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:36.437473 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:36.442500 (+ 5027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:36.442527 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:36.442546 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:36.442598 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:36.446263 (+ 3665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:36.447397 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:36.447406 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:36.447408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:36.447615 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:36.448142 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:36.448145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:36.448147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:36.448609 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:36.448876 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:36.449341 (+ 465us) server_negotiation.cc:299] Negotiation successful
0928 16:50:36.449567 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:50:36.450687 17444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:36.618891 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 17312
I20260928 16:50:36.641501 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 17453
Sep 28 16:50:36 dist-test-slave-wzcb krb5kdc[17194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.93.2@KRBTEST.COM
I20260928 16:50:36.661659 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:36.642458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35068 (local address 127.9.93.2:33729)
0928 16:50:36.643024 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:36.643028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:36.643047 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:50:36.643113 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:36.643132 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:36.643209 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:36.643345 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:36.644993 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:36.645799 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:36.647969 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:36.648465 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:36.648581 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:36.654037 (+ 5456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:36.654061 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:36.654081 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:36.654138 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:36.657775 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:36.658858 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:36.658870 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:36.658873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:36.659133 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:36.659690 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:36.659695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:36.659697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:36.660179 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:36.660568 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:36.660952 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 16:50:36.661276 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":325,"threads_started":1}
I20260928 16:50:36.662576 17585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:36.768697 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 17453
I20260928 16:50:36.787249 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 17594
Sep 28 16:50:36 dist-test-slave-wzcb krb5kdc[17194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.93.3@KRBTEST.COM
I20260928 16:50:36.805461 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:36.788218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37142 (local address 127.9.93.3:37881)
0928 16:50:36.788833 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:36.788839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:36.788858 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:50:36.788925 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:36.788929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:36.788997 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:36.789105 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:36.790605 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:36.791603 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:36.793266 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:36.793836 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:36.793959 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:36.798568 (+ 4609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:36.798591 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:36.798612 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:36.798661 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:36.801895 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:36.802867 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:36.802878 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:36.802880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:36.803056 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:36.803591 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:36.803597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:36.803599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:36.804080 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:36.804431 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:36.804766 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 16:50:36.805065 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":356,"threads_started":1}
I20260928 16:50:36.806234 17725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:36.948205 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 17594
I20260928 16:50:36.968149 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 17207
I20260928 16:50:36.969532 17280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:37.093159 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 17207
2026-09-28T16:50:37Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6209 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:50:37 dist-test-slave-wzcb krb5kdc[17867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:50:37 dist-test-slave-wzcb krb5kdc[17867](info): set up 2 sockets
Sep 28 16:50:37 dist-test-slave-wzcb krb5kdc[17867](info): commencing operation
krb5kdc: starting...
W20260928 16:50:38.146658 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:50:38 dist-test-slave-wzcb krb5kdc[17867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:50:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:50:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:50:38.400035 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:41923
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:43431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:41923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:50:38.759140 17880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:38.759536 17880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:38.759683 17880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:38.767565 17880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:50:38.767696 17880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:38.767776 17880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:38.767858 17880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:50:38.767923 17880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:50:38.778749 17880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:43431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:41923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:41923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:38.780745 17880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:38.782655 17880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:38.795811 17885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:38.795852 17888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:38.795849 17886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:38.797374 17880 server_base.cc:1034] running on GCE node
I20260928 16:50:38.798323 17880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:38.800081 17880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:38.801342 17880 hybrid_clock.cc:648] HybridClock initialized: now 1790614238801293 us; error 37 us; skew 500 ppm
I20260928 16:50:38.801752 17880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T16:50:38Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1780 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:50:38 dist-test-slave-wzcb krb5kdc[17894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:50:38 dist-test-slave-wzcb krb5kdc[17894](info): set up 2 sockets
Sep 28 16:50:38 dist-test-slave-wzcb krb5kdc[17894](info): commencing operation
krb5kdc: starting...
W20260928 16:50:39.970985 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:50:40 dist-test-slave-wzcb krb5kdc[17894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:50:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:50:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:50:40.358100 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:33515
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:42831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:33515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:50:40.988310 17908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:40.988850 17908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:40.989001 17908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:41.002200 17908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:50:41.002367 17908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:41.002457 17908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:41.002543 17908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:50:41.002669 17908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:50:41.020613 17908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:42831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:33515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:33515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:41.023353 17908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:41.026378 17908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:41.048416 17913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:41.048281 17916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:41.048545 17914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:41.050459 17908 server_base.cc:1034] running on GCE node
I20260928 16:50:41.052316 17908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:41.055397 17908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:41.058962 17908 hybrid_clock.cc:648] HybridClock initialized: now 1790614241058828 us; error 73 us; skew 500 ppm
I20260928 16:50:41.059721 17908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:41 dist-test-slave-wzcb krb5kdc[17894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614241, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:41.069716 17908 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:50:41.072947 17908 webserver.cc:460] Webserver started at http://127.9.93.62:45515/ using document root <none> and password file <none>
I20260928 16:50:41.074544 17908 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:41.074828 17908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:41.075610 17908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:41.082181 17908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "c0d2a3d6ba70489590dfec62ba6893bb"
format_stamp: "Formatted at 2026-09-28 16:50:41 on dist-test-slave-wzcb"
server_key: "7c57e676a1195df09e31251fc45e5cae"
server_key_iv: "e45fa1f6ff05bae0eb135f2be3133959"
server_key_version: "encryptionkey@0"
I20260928 16:50:41.084426 17908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "c0d2a3d6ba70489590dfec62ba6893bb"
format_stamp: "Formatted at 2026-09-28 16:50:41 on dist-test-slave-wzcb"
server_key: "7c57e676a1195df09e31251fc45e5cae"
server_key_iv: "e45fa1f6ff05bae0eb135f2be3133959"
server_key_version: "encryptionkey@0"
I20260928 16:50:41.097033 17908 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.010s sys 0.001s
I20260928 16:50:41.104307 17923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:41.107272 17908 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.001s
I20260928 16:50:41.107712 17908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "c0d2a3d6ba70489590dfec62ba6893bb"
format_stamp: "Formatted at 2026-09-28 16:50:41 on dist-test-slave-wzcb"
server_key: "7c57e676a1195df09e31251fc45e5cae"
server_key_iv: "e45fa1f6ff05bae0eb135f2be3133959"
server_key_version: "encryptionkey@0"
I20260928 16:50:41.108127 17908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:41.135483 17908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:41.140259 17908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:41.140578 17908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:41.159965 17908 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:33515
I20260928 16:50:41.160014 17974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:33515 every 8 connection(s)
I20260928 16:50:41.161636 17908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:50:41.166145 17975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:50:41.168876 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 17908
I20260928 16:50:41.169093 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:50:41.169785 9588 external_mini_cluster.cc:1428] Setting key 567dcc5c8b3377dab41b0f35ee747684
I20260928 16:50:41.180238 17975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb: Bootstrap starting.
I20260928 16:50:41.184386 17975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb: Neither blocks nor log segments found. Creating new log.
I20260928 16:50:41.185566 17975 log.cc:824] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:50:41 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:50:41.189301 17975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb: No bootstrap required, opened a new log
I20260928 16:50:41.198002 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:41.172185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34730 (local address 127.9.93.62:33515)
0928 16:50:41.173115 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:41.173134 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:41.173194 (+ 60us) server_negotiation.cc:407] Connection header received
0928 16:50:41.175649 (+ 2455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:41.175686 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:41.176343 (+ 657us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:41.176879 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:41.178617 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:41.180673 (+ 2056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:41.182211 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:41.183297 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:41.183555 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:41.188219 (+ 4664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:41.188232 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:41.188254 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:41.188304 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:41.192514 (+ 4210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:41.193426 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:41.193441 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:41.193452 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:41.193693 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:41.194088 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:41.194094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:41.194097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:41.194940 (+ 843us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:41.195264 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:41.195790 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 16:50:41.196336 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":382,"threads_started":1}
I20260928 16:50:41.199966 17975 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d2a3d6ba70489590dfec62ba6893bb" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33515 } }
I20260928 16:50:41.200418 17975 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:50:41.200536 17975 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d2a3d6ba70489590dfec62ba6893bb, State: Initialized, Role: FOLLOWER
I20260928 16:50:41.201403 17975 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d2a3d6ba70489590dfec62ba6893bb" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33515 } }
I20260928 16:50:41.201679 17975 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:50:41.201845 17975 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:50:41.202051 17975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:50:41.205145 17975 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d2a3d6ba70489590dfec62ba6893bb" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33515 } }
I20260928 16:50:41.205734 17975 leader_election.cc:302] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0d2a3d6ba70489590dfec62ba6893bb; no voters:
I20260928 16:50:41.206367 17975 leader_election.cc:288] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:50:41.206569 17980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:50:41.207269 17980 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [term 1 LEADER]: Becoming Leader. State: Replica: c0d2a3d6ba70489590dfec62ba6893bb, State: Running, Role: LEADER
I20260928 16:50:41.207898 17980 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d2a3d6ba70489590dfec62ba6893bb" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33515 } }
I20260928 16:50:41.208779 17975 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:50:41.210949 17981 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0d2a3d6ba70489590dfec62ba6893bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d2a3d6ba70489590dfec62ba6893bb" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33515 } } }
I20260928 16:50:41.211432 17981 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:41.212350 17987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:50:41.212227 17982 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0d2a3d6ba70489590dfec62ba6893bb. Latest consensus state: current_term: 1 leader_uuid: "c0d2a3d6ba70489590dfec62ba6893bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d2a3d6ba70489590dfec62ba6893bb" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33515 } } }
I20260928 16:50:41.213810 17982 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:41.219753 17987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:50:41.232704 17987 catalog_manager.cc:1347] Generated new cluster ID: 232d89b7cbaa46bf95ddbff4fb70934d
I20260928 16:50:41.232836 17987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:50:41.250327 17987 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:50:41.251824 17987 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:50:41.272795 17987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb: Generated new TSK 0
I20260928 16:50:41.273913 17987 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
I20260928 16:50:41.340265 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33515
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:42831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:50:41.703353 18001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:41.703696 18001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:41.703830 18001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:41.712085 18001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:41.712225 18001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:41.712393 18001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:50:41.723839 18001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:42831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:41.725432 18001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:41.726985 18001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:41.742067 18006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:41.741976 18009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:41.742019 18007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:41.743018 18001 server_base.cc:1034] running on GCE node
I20260928 16:50:41.743825 18001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:41.745020 18001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:41.746258 18001 hybrid_clock.cc:648] HybridClock initialized: now 1790614241746181 us; error 63 us; skew 500 ppm
I20260928 16:50:41.746634 18001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:41 dist-test-slave-wzcb krb5kdc[17894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614241, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:41.750259 18001 init.cc:377] Logged in from keytab as kudu/127.9.93.1@KRBTEST.COM (short username kudu)
I20260928 16:50:41.752060 18001 webserver.cc:460] Webserver started at http://127.9.93.1:33855/ using document root <none> and password file <none>
I20260928 16:50:41.752835 18001 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:41.752952 18001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:41.753254 18001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:41.756731 18001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "cc1509c464ab46dd8ce9c7078818a90b"
format_stamp: "Formatted at 2026-09-28 16:50:41 on dist-test-slave-wzcb"
server_key: "4e4899c24a2828fae8c787f46b91ea24"
server_key_iv: "c6fccb385dafa4a402806e7a9adc10bb"
server_key_version: "encryptionkey@0"
I20260928 16:50:41.757575 18001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "cc1509c464ab46dd8ce9c7078818a90b"
format_stamp: "Formatted at 2026-09-28 16:50:41 on dist-test-slave-wzcb"
server_key: "4e4899c24a2828fae8c787f46b91ea24"
server_key_iv: "c6fccb385dafa4a402806e7a9adc10bb"
server_key_version: "encryptionkey@0"
I20260928 16:50:41.763585 18001 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 16:50:41.767941 18016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:41.769739 18001 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:50:41.769922 18001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "cc1509c464ab46dd8ce9c7078818a90b"
format_stamp: "Formatted at 2026-09-28 16:50:41 on dist-test-slave-wzcb"
server_key: "4e4899c24a2828fae8c787f46b91ea24"
server_key_iv: "c6fccb385dafa4a402806e7a9adc10bb"
server_key_version: "encryptionkey@0"
I20260928 16:50:41.770141 18001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:41.801455 18001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:41.806298 18001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:41.806608 18001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:41.807814 18001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:50:41.809482 18001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:41.809561 18001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:41.809664 18001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:41.809710 18001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:41.845191 18001 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:38551
I20260928 16:50:41.845233 18129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:38551 every 8 connection(s)
I20260928 16:50:41.846647 18001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:50:41.856446 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 18001
I20260928 16:50:41.856649 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:50:41.857508 9588 external_mini_cluster.cc:1428] Setting key 6462b3e8600202d0c2edadde41bbc00e
Sep 28 16:50:41 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614241, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:50:41.870198 18132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:41.849722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:51719 (local address 127.9.93.62:33515)
0928 16:50:41.850265 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:41.850272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:41.851581 (+ 1309us) server_negotiation.cc:407] Connection header received
0928 16:50:41.853685 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:41.853691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:41.853774 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:41.853892 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:41.855984 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:41.856875 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:41.859176 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:41.859743 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:41.859865 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:41.864159 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:41.864165 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:41.864169 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:41.864206 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:41.866752 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:41.867686 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:41.867691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:41.867694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:41.867844 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:41.868305 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:41.868310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:41.868312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:41.868609 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:41.868748 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:41.869559 (+ 811us) server_negotiation.cc:299] Negotiation successful
0928 16:50:41.869814 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":258,"threads_started":1}
I20260928 16:50:41.871366 18133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:41.850481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.1:51719)
0928 16:50:41.851371 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:41.851449 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:41.853332 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:41.854210 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:41.854221 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:41.854856 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:41.855707 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:41.855734 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:41.857066 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:41.857072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:41.858992 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:41.859008 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:41.860021 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:41.860030 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:41.860292 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:41.861062 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:41.861093 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:41.863895 (+ 2802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:41.866921 (+ 3026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:41.866933 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:41.866949 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:41.867548 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:41.867960 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:41.867965 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:41.867968 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:41.868184 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:41.868742 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:41.868750 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:41.869148 (+ 398us) client_negotiation.cc:769] Sending connection context
0928 16:50:41.869423 (+ 275us) client_negotiation.cc:240] Negotiation successful
0928 16:50:41.870065 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":384,"threads_started":1}
I20260928 16:50:41.873360 18130 heartbeater.cc:347] Connected to a master server at 127.9.93.62:33515
I20260928 16:50:41.873739 18130 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:41.874676 18130 heartbeater.cc:511] Master 127.9.93.62:33515 requested a full tablet report, sending...
I20260928 16:50:41.877435 17940 ts_manager.cc:194] Registered new tserver with Master: cc1509c464ab46dd8ce9c7078818a90b (127.9.93.1:38551)
I20260928 16:50:41.879746 17940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.1@KRBTEST.COM'} at 127.9.93.1:51719
WARNING: no policy specified for kudu/127.9.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
I20260928 16:50:41.915858 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33515
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:42831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:50:42.283615 18136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:42.283952 18136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:42.284057 18136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:42.292425 18136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:42.292608 18136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:42.292759 18136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:50:42.303683 18136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:42831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:42.305214 18136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:42.306701 18136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:42.321401 18144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:42.321390 18141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:42.321831 18142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:42.321797 18136 server_base.cc:1034] running on GCE node
I20260928 16:50:42.322817 18136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:42.323980 18136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:42.325208 18136 hybrid_clock.cc:648] HybridClock initialized: now 1790614242325148 us; error 54 us; skew 500 ppm
I20260928 16:50:42.325590 18136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:42 dist-test-slave-wzcb krb5kdc[17894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614242, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:42.329066 18136 init.cc:377] Logged in from keytab as kudu/127.9.93.2@KRBTEST.COM (short username kudu)
I20260928 16:50:42.330839 18136 webserver.cc:460] Webserver started at http://127.9.93.2:44101/ using document root <none> and password file <none>
I20260928 16:50:42.331712 18136 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:42.331846 18136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:42.332186 18136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:42.335664 18136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "86a269d9e14c4584b575779bb5d45450"
format_stamp: "Formatted at 2026-09-28 16:50:42 on dist-test-slave-wzcb"
server_key: "4b919842ec8bf62550d7e17708515d0a"
server_key_iv: "e09e2959901be3b3c54a26644528167d"
server_key_version: "encryptionkey@0"
I20260928 16:50:42.336519 18136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "86a269d9e14c4584b575779bb5d45450"
format_stamp: "Formatted at 2026-09-28 16:50:42 on dist-test-slave-wzcb"
server_key: "4b919842ec8bf62550d7e17708515d0a"
server_key_iv: "e09e2959901be3b3c54a26644528167d"
server_key_version: "encryptionkey@0"
I20260928 16:50:42.342550 18136 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:50:42.346944 18151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:42.348732 18136 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:50:42.348948 18136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "86a269d9e14c4584b575779bb5d45450"
format_stamp: "Formatted at 2026-09-28 16:50:42 on dist-test-slave-wzcb"
server_key: "4b919842ec8bf62550d7e17708515d0a"
server_key_iv: "e09e2959901be3b3c54a26644528167d"
server_key_version: "encryptionkey@0"
I20260928 16:50:42.349185 18136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:42.365082 18136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:42.369846 18136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:42.370165 18136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:42.371398 18136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:50:42.373144 18136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:42.373234 18136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:42.373361 18136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:42.373427 18136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:42.407090 18136 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:37639
I20260928 16:50:42.407162 18263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:37639 every 8 connection(s)
I20260928 16:50:42.408694 18136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:50:42.412314 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 18136
I20260928 16:50:42.412514 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:50:42.413136 9588 external_mini_cluster.cc:1428] Setting key 61bbb268c6a1dc0f7afdcb5d227b7720
Sep 28 16:50:42 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614242, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:50:42.431697 18266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:42.411614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:47879 (local address 127.9.93.62:33515)
0928 16:50:42.412088 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:42.412093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:42.413348 (+ 1255us) server_negotiation.cc:407] Connection header received
0928 16:50:42.415658 (+ 2310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:42.415664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:42.415758 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:42.415865 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:42.417896 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:42.418661 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:42.420397 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:42.420877 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:42.421002 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:42.424984 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:42.424991 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:42.424995 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:42.425032 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:42.427830 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:42.428872 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:42.428877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:42.428879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:42.429040 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:42.429585 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:42.429587 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:42.429589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:42.429869 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:42.429986 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:42.431057 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0928 16:50:42.431268 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":255,"threads_started":1}
I20260928 16:50:42.433161 18267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:42.412253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.2:47879)
0928 16:50:42.413160 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:42.413209 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:42.415348 (+ 2139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:42.416261 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:42.416270 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:42.416927 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:42.417707 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:42.417730 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:42.418830 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:42.418834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:42.420238 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:42.420252 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:42.421140 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:42.421149 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:42.421408 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:42.422106 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:42.422131 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:42.424682 (+ 2551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:42.428013 (+ 3331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:42.428023 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:42.428045 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:42.428693 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:42.429169 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:42.429174 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:42.429178 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:42.429467 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:42.429996 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:42.430004 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:42.430545 (+ 541us) client_negotiation.cc:769] Sending connection context
0928 16:50:42.430898 (+ 353us) client_negotiation.cc:240] Negotiation successful
0928 16:50:42.431560 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":376,"threads_started":1}
I20260928 16:50:42.434863 18264 heartbeater.cc:347] Connected to a master server at 127.9.93.62:33515
I20260928 16:50:42.435269 18264 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:42.436365 18264 heartbeater.cc:511] Master 127.9.93.62:33515 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.93.3@KRBTEST.COM; defaulting to no policy
I20260928 16:50:42.438465 17940 ts_manager.cc:194] Registered new tserver with Master: 86a269d9e14c4584b575779bb5d45450 (127.9.93.2:37639)
I20260928 16:50:42.439357 17940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.2@KRBTEST.COM'} at 127.9.93.2:47879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
I20260928 16:50:42.468412 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33515
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:42831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:50:42.837914 18270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:42.838295 18270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:42.838405 18270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:42.846839 18270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:42.846990 18270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:42.847185 18270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:50:42.858646 18270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:42831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:42.860291 18270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:42.861886 18270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:42.877630 18275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:42.877506 18276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:42.877503 18278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:42.878575 18270 server_base.cc:1034] running on GCE node
I20260928 16:50:42.879446 18270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:42.880702 18270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:42.881970 18270 hybrid_clock.cc:648] HybridClock initialized: now 1790614242881900 us; error 68 us; skew 500 ppm
I20260928 16:50:42.882377 18270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:50:42.883884 18130 heartbeater.cc:503] Master 127.9.93.62:33515 was elected leader, sending a full tablet report...
Sep 28 16:50:42 dist-test-slave-wzcb krb5kdc[17894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614242, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:42.886581 18270 init.cc:377] Logged in from keytab as kudu/127.9.93.3@KRBTEST.COM (short username kudu)
I20260928 16:50:42.888430 18270 webserver.cc:460] Webserver started at http://127.9.93.3:41885/ using document root <none> and password file <none>
I20260928 16:50:42.889257 18270 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:42.889379 18270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:42.889715 18270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:50:42.893392 18270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "609077d1f9194fb9aa6002c180b7c834"
format_stamp: "Formatted at 2026-09-28 16:50:42 on dist-test-slave-wzcb"
server_key: "7e5d0bfe6aab3f31537a5e5aab76d9dc"
server_key_iv: "46f9e96e2eb8a50b12392cc636484412"
server_key_version: "encryptionkey@0"
I20260928 16:50:42.894392 18270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "609077d1f9194fb9aa6002c180b7c834"
format_stamp: "Formatted at 2026-09-28 16:50:42 on dist-test-slave-wzcb"
server_key: "7e5d0bfe6aab3f31537a5e5aab76d9dc"
server_key_iv: "46f9e96e2eb8a50b12392cc636484412"
server_key_version: "encryptionkey@0"
I20260928 16:50:42.901054 18270 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 16:50:42.905550 18285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:42.907459 18270 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:50:42.907713 18270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "609077d1f9194fb9aa6002c180b7c834"
format_stamp: "Formatted at 2026-09-28 16:50:42 on dist-test-slave-wzcb"
server_key: "7e5d0bfe6aab3f31537a5e5aab76d9dc"
server_key_iv: "46f9e96e2eb8a50b12392cc636484412"
server_key_version: "encryptionkey@0"
I20260928 16:50:42.907966 18270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:42.924183 18270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:42.929163 18270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:42.929482 18270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:42.930645 18270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:50:42.932412 18270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:42.932498 18270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:42.932605 18270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:42.932658 18270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:42.967609 18270 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:43933
I20260928 16:50:42.967636 18399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:43933 every 8 connection(s)
I20260928 16:50:42.969282 18270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:50:42.975275 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 18270
I20260928 16:50:42.975498 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:50:42.976154 9588 external_mini_cluster.cc:1428] Setting key 547721d44081151b79507470815cf3f6
Sep 28 16:50:42 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614242, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:50:42.994336 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:42.972344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:44497 (local address 127.9.93.62:33515)
0928 16:50:42.972863 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:42.972868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:42.973997 (+ 1129us) server_negotiation.cc:407] Connection header received
0928 16:50:42.976498 (+ 2501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:42.976533 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:42.976714 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:42.976879 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:42.979179 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:42.980229 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:42.982107 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:42.982620 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:42.982761 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:42.987121 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:42.987140 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:42.987144 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:42.987199 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:42.990268 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:42.991632 (+ 1364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:42.991638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:42.991642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:42.991836 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:42.992443 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:42.992448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:42.992450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:42.992838 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:42.992991 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:42.993815 (+ 824us) server_negotiation.cc:299] Negotiation successful
0928 16:50:42.994024 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":253,"threads_started":1}
I20260928 16:50:42.995532 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:42.972997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.3:44497)
0928 16:50:42.973824 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:42.973866 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:42.976099 (+ 2233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:42.977195 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:42.977205 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:42.978005 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:42.978966 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:42.979000 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:42.980414 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:42.980418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:42.981896 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:42.981915 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:42.982952 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:42.982959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:42.983234 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:42.984055 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:42.984083 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:42.986791 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:50:42.990591 (+ 3800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:42.990605 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:42.990629 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:42.991452 (+ 823us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:42.991984 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:50:42.991990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:50:42.991993 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:50:42.992294 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:50:42.992976 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:50:42.992984 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:50:42.993423 (+ 439us) client_negotiation.cc:769] Sending connection context
0928 16:50:42.993692 (+ 269us) client_negotiation.cc:240] Negotiation successful
0928 16:50:42.994222 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":322,"threads_started":1}
I20260928 16:50:42.997219 18400 heartbeater.cc:347] Connected to a master server at 127.9.93.62:33515
I20260928 16:50:42.997577 18400 heartbeater.cc:464] Registering TS with master...
I20260928 16:50:42.998441 18400 heartbeater.cc:511] Master 127.9.93.62:33515 requested a full tablet report, sending...
I20260928 16:50:43.000334 17940 ts_manager.cc:194] Registered new tserver with Master: 609077d1f9194fb9aa6002c180b7c834 (127.9.93.3:43933)
I20260928 16:50:43.001245 17940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.3@KRBTEST.COM'} at 127.9.93.3:44497
I20260928 16:50:43.005741 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.1.keytab.
Entry for principal oryx/127.9.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.1.keytab.
WARNING: no policy specified for oryx/127.9.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.2.keytab.
Entry for principal oryx/127.9.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.2.keytab.
WARNING: no policy specified for oryx/127.9.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.3.keytab.
Entry for principal oryx/127.9.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.3.keytab.
I20260928 16:50:43.168288 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 18001
Sep 28 16:50:43 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.1@KRBTEST.COM
I20260928 16:50:43.187605 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:43.169705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32904 (local address 127.9.93.1:38551)
0928 16:50:43.170518 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:43.170527 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:43.170559 (+ 32us) server_negotiation.cc:407] Connection header received
0928 16:50:43.170642 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:43.170649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:43.170908 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:43.171077 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:43.172672 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:43.173846 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:43.175343 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:43.176140 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:43.176273 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:43.180859 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:43.180878 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:43.180896 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:43.180952 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:43.184191 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:43.185066 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:43.185075 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:43.185078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:43.185237 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:43.185739 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:43.185744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:43.185746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:43.186207 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:43.186492 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:43.186865 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 16:50:43.187155 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":450,"threads_started":1}
I20260928 16:50:43.189106 18125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:43.281785 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 18001
I20260928 16:50:43.297971 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 18136
Sep 28 16:50:43 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.2@KRBTEST.COM
I20260928 16:50:43.316654 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:43.299034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38320 (local address 127.9.93.2:37639)
0928 16:50:43.299736 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:43.299745 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:43.299775 (+ 30us) server_negotiation.cc:407] Connection header received
0928 16:50:43.299849 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:43.299856 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:43.300105 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:43.300251 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:43.301615 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:43.302688 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:43.304222 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:43.304925 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:43.305051 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:43.309729 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:43.309749 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:43.309767 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:43.309815 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:43.313179 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:43.314070 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:43.314079 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:43.314082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:43.314245 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:43.314718 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:43.314722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:43.314724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:43.315150 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:43.315532 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:43.315926 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 16:50:43.316202 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":375,"threads_started":1}
I20260928 16:50:43.318439 18259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:43.406128 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 18136
I20260928 16:50:43.421397 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 18270
Sep 28 16:50:43 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.3@KRBTEST.COM
I20260928 16:50:43.440263 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:43.422638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44130 (local address 127.9.93.3:43933)
0928 16:50:43.423241 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:43.423250 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:43.423283 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:50:43.423387 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:43.423396 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:43.423701 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:43.423842 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:43.425278 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:43.426417 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:43.428141 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:43.428878 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:43.429020 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:43.433831 (+ 4811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:43.433860 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:43.433879 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:43.433930 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:43.437179 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:43.438008 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:43.438017 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:43.438020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:43.438172 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:43.438628 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:43.438632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:43.438634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:43.439066 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:43.439396 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:43.439672 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 16:50:43.439918 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":271,"threads_started":1}
I20260928 16:50:43.441802 18395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:43.531383 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 18270
I20260928 16:50:43.545499 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 17908
I20260928 16:50:43.546977 17970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:50:43.655710 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 17908
I20260928 16:50:43.674590 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:33515
--webserver_interface=127.9.93.62
--webserver_port=45515
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:42831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:33515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:50:44.039799 18418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:44.040143 18418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:44.040238 18418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:44.047822 18418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:50:44.047959 18418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:44.048007 18418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:44.048059 18418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:50:44.048103 18418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:50:44.059232 18418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:42831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:33515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:33515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=45515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:44.060880 18418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:44.062418 18418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:44.075749 18423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:44.075724 18424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:44.075683 18426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:44.077340 18418 server_base.cc:1034] running on GCE node
I20260928 16:50:44.078358 18418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:44.080147 18418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:44.081467 18418 hybrid_clock.cc:648] HybridClock initialized: now 1790614244081387 us; error 62 us; skew 500 ppm
I20260928 16:50:44.081919 18418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:44 dist-test-slave-wzcb krb5kdc[17894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614244, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:44.085667 18418 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:50:44.087467 18418 webserver.cc:460] Webserver started at http://127.9.93.62:45515/ using document root <none> and password file <none>
I20260928 16:50:44.088294 18418 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:44.088454 18418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:44.095127 18418 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:50:44.098727 18433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:44.100562 18418 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:50:44.100793 18418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "c0d2a3d6ba70489590dfec62ba6893bb"
format_stamp: "Formatted at 2026-09-28 16:50:41 on dist-test-slave-wzcb"
server_key: "7c57e676a1195df09e31251fc45e5cae"
server_key_iv: "e45fa1f6ff05bae0eb135f2be3133959"
server_key_version: "encryptionkey@0"
I20260928 16:50:44.101943 18418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:44.122668 18418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:44.128167 18418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:44.128504 18418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:44.148763 18418 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:33515
I20260928 16:50:44.148777 18484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:33515 every 8 connection(s)
I20260928 16:50:44.150800 18418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:50:44.151916 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 18418
I20260928 16:50:44.152897 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:38551
--local_ip_for_outbound_sockets=127.9.93.1
--tserver_master_addrs=127.9.93.62:33515
--webserver_port=33855
--webserver_interface=127.9.93.1
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:42831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.93.1
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:50:44.159127 18485 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:50:44.163902 18485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb: Bootstrap starting.
I20260928 16:50:44.194325 18485 log.cc:824] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb: Log is configured to *not* fsync() on all Append() calls
I20260928 16:50:44.208253 18485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:50:44.209007 18485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb: Bootstrap complete.
I20260928 16:50:44.222975 18485 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d2a3d6ba70489590dfec62ba6893bb" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33515 } }
I20260928 16:50:44.224341 18485 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d2a3d6ba70489590dfec62ba6893bb, State: Initialized, Role: FOLLOWER
I20260928 16:50:44.225215 18485 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d2a3d6ba70489590dfec62ba6893bb" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33515 } }
I20260928 16:50:44.225540 18485 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:50:44.225759 18485 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:50:44.225997 18485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:50:44.233040 18485 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d2a3d6ba70489590dfec62ba6893bb" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33515 } }
I20260928 16:50:44.233855 18485 leader_election.cc:302] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0d2a3d6ba70489590dfec62ba6893bb; no voters:
I20260928 16:50:44.234640 18485 leader_election.cc:288] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 16:50:44.234841 18489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [term 2 FOLLOWER]: Leader election won for term 2
I20260928 16:50:44.236141 18489 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [term 2 LEADER]: Becoming Leader. State: Replica: c0d2a3d6ba70489590dfec62ba6893bb, State: Running, Role: LEADER
I20260928 16:50:44.236785 18489 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d2a3d6ba70489590dfec62ba6893bb" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33515 } }
I20260928 16:50:44.237212 18485 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:50:44.239588 18490 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c0d2a3d6ba70489590dfec62ba6893bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d2a3d6ba70489590dfec62ba6893bb" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33515 } } }
I20260928 16:50:44.239521 18491 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0d2a3d6ba70489590dfec62ba6893bb. Latest consensus state: current_term: 2 leader_uuid: "c0d2a3d6ba70489590dfec62ba6893bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d2a3d6ba70489590dfec62ba6893bb" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33515 } } }
I20260928 16:50:44.239916 18490 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:44.239913 18491 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb [sys.catalog]: This master's current role is: LEADER
I20260928 16:50:44.240913 18495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:50:44.258988 18495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:50:44.261708 18495 catalog_manager.cc:1259] Loaded cluster ID: 232d89b7cbaa46bf95ddbff4fb70934d
I20260928 16:50:44.261926 18495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:50:44.266208 18495 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:50:44.268018 18495 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c0d2a3d6ba70489590dfec62ba6893bb: Loaded TSK: 0
I20260928 16:50:44.270720 18495 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:50:44.571586 18487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:44.571928 18487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:44.572086 18487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:44.580411 18487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:44.580574 18487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:44.580749 18487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:50:44.592298 18487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:42831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.1.keytab
--principal=oryx/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:38551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=33855
--tserver_master_addrs=127.9.93.62:33515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:44.593928 18487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:44.595597 18487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:44.611374 18512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:44.611747 18514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:44.611490 18511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:44.613262 18487 server_base.cc:1034] running on GCE node
I20260928 16:50:44.614048 18487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:44.615260 18487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:44.616561 18487 hybrid_clock.cc:648] HybridClock initialized: now 1790614244616487 us; error 64 us; skew 500 ppm
I20260928 16:50:44.616928 18487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:44 dist-test-slave-wzcb krb5kdc[17894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614244, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:44.620759 18487 init.cc:377] Logged in from keytab as oryx/127.9.93.1@KRBTEST.COM (short username oryx)
I20260928 16:50:44.622610 18487 webserver.cc:460] Webserver started at http://127.9.93.1:33855/ using document root <none> and password file <none>
I20260928 16:50:44.623473 18487 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:44.623637 18487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:44.630018 18487 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:50:44.633538 18521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:44.635427 18487 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:50:44.635654 18487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "cc1509c464ab46dd8ce9c7078818a90b"
format_stamp: "Formatted at 2026-09-28 16:50:41 on dist-test-slave-wzcb"
server_key: "4e4899c24a2828fae8c787f46b91ea24"
server_key_iv: "c6fccb385dafa4a402806e7a9adc10bb"
server_key_version: "encryptionkey@0"
I20260928 16:50:44.636869 18487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:44.658485 18487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:44.663472 18487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:44.663794 18487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:44.665050 18487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:50:44.666800 18487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:44.666920 18487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:44.667064 18487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:44.667169 18487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:44.703159 18487 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:38551
I20260928 16:50:44.703225 18635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:38551 every 8 connection(s)
I20260928 16:50:44.704968 18487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:50:44.705392 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 18487
I20260928 16:50:44.706243 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:37639
--local_ip_for_outbound_sockets=127.9.93.2
--tserver_master_addrs=127.9.93.62:33515
--webserver_port=44101
--webserver_interface=127.9.93.2
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:42831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.93.2
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:50:44 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:44 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:44.728912 18639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:44.708899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.1:43497)
0928 16:50:44.710286 (+ 1387us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:44.710363 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:44.713057 (+ 2694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:44.715183 (+ 2126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:44.715197 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:44.716144 (+ 947us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:44.716888 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:44.716914 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:44.718891 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:44.718896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:44.720487 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:44.720509 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:44.721812 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:44.721821 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:44.722154 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:44.722894 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:44.722930 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:44.727263 (+ 4333us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1018,"thread_start_us":766,"threads_started":1}
W20260928 16:50:44.729112 18638 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:44.708852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:43497 (local address 127.9.93.62:33515)
0928 16:50:44.709851 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:44.709873 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:44.710545 (+ 672us) server_negotiation.cc:407] Connection header received
0928 16:50:44.713890 (+ 3345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:44.713908 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:44.714403 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:44.714838 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:44.717195 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:44.718645 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:44.720692 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:44.721500 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:44.721715 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:44.727829 (+ 6114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.1:43497: BlockingRecv error: recv got EOF from 127.9.93.1:43497 (error 108)
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":328,"threads_started":1}
W20260928 16:50:44.730134 18636 heartbeater.cc:644] Failed to heartbeat to 127.9.93.62:33515 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.93.62:33515: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Sep 28 16:50:44 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:44 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:44.740155 18639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:44.730919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.1:41039)
0928 16:50:44.731135 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:44.731167 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:44.731289 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:44.731794 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:44.731799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:44.732220 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:44.732399 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:44.732410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:44.733546 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:44.733550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:44.734576 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:44.734591 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:44.735381 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:44.735387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:44.735578 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:44.736242 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:44.736264 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:44.739827 (+ 3563us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260928 16:50:44.740583 18638 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:44.731038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:41039 (local address 127.9.93.62:33515)
0928 16:50:44.731481 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:44.731486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:44.731504 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:50:44.731567 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:44.731572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:44.731639 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:44.731731 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:44.732580 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:44.733367 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:44.734741 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:44.735182 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:44.735319 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:44.740129 (+ 4810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.1:41039: BlockingRecv error: recv got EOF from 127.9.93.1:41039 (error 108)
Metrics: {"server-negotiator.queue_time_us":313}
Sep 28 16:50:44 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:44 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:44.750733 18639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:44.741172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.1:44735)
0928 16:50:44.741371 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:44.741393 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:44.741494 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:44.741920 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:44.741925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:44.742304 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:44.742560 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:44.742574 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:44.744060 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:44.744065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:44.745058 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:44.745073 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:44.746026 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:44.746042 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:44.746229 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:44.746914 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:44.746936 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:44.750455 (+ 3519us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 16:50:44.751040 18636 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:50:44.751065 18638 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:44.741290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:44735 (local address 127.9.93.62:33515)
0928 16:50:44.741485 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:44.741494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:44.741516 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:50:44.741675 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:44.741681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:44.741784 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:44.741915 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:44.742733 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:44.743869 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:44.745204 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:44.745809 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:44.745940 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:44.750789 (+ 4849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.1:44735: BlockingRecv error: recv got EOF from 127.9.93.1:44735 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 16:50:45.089902 18640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:45.090255 18640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:45.090395 18640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:45.098568 18640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:45.098716 18640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:45.098884 18640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:50:45.110158 18640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:42831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.2.keytab
--principal=oryx/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:37639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=44101
--tserver_master_addrs=127.9.93.62:33515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:45.111845 18640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:45.113472 18640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:45.128942 18646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:45.129004 18648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:45.129659 18645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:45.129902 18640 server_base.cc:1034] running on GCE node
I20260928 16:50:45.130699 18640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:45.131933 18640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:45.133164 18640 hybrid_clock.cc:648] HybridClock initialized: now 1790614245133097 us; error 57 us; skew 500 ppm
I20260928 16:50:45.133529 18640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:45 dist-test-slave-wzcb krb5kdc[17894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614245, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:45.137600 18640 init.cc:377] Logged in from keytab as oryx/127.9.93.2@KRBTEST.COM (short username oryx)
I20260928 16:50:45.139617 18640 webserver.cc:460] Webserver started at http://127.9.93.2:44101/ using document root <none> and password file <none>
I20260928 16:50:45.140512 18640 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:45.140678 18640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:45.147800 18640 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.007s
I20260928 16:50:45.151521 18655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:45.153513 18640 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:50:45.153750 18640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "86a269d9e14c4584b575779bb5d45450"
format_stamp: "Formatted at 2026-09-28 16:50:42 on dist-test-slave-wzcb"
server_key: "4b919842ec8bf62550d7e17708515d0a"
server_key_iv: "e09e2959901be3b3c54a26644528167d"
server_key_version: "encryptionkey@0"
I20260928 16:50:45.154975 18640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:45.170185 18640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:45.175593 18640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:45.175917 18640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:45.177150 18640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:50:45.178798 18640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:45.178922 18640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:45.179047 18640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:45.179126 18640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:45.221400 18640 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:37639
I20260928 16:50:45.221458 18768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:37639 every 8 connection(s)
I20260928 16:50:45.223160 18640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:50:45.228189 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 18640
I20260928 16:50:45.229017 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:43933
--local_ip_for_outbound_sockets=127.9.93.3
--tserver_master_addrs=127.9.93.62:33515
--webserver_port=41885
--webserver_interface=127.9.93.3
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:42831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.93.3
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:50:45 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:45 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 16:50:45.244474 18638 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:45.226380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:55549 (local address 127.9.93.62:33515)
0928 16:50:45.226611 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:45.226616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:45.228174 (+ 1558us) server_negotiation.cc:407] Connection header received
0928 16:50:45.231270 (+ 3096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:45.231276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:45.231403 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:45.231502 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:45.234140 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.234947 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.237518 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.238068 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.238190 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:45.244200 (+ 6010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.2:55549: BlockingRecv error: recv got EOF from 127.9.93.2:55549 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
I20260928 16:50:45.245160 18771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:45.226873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.2:55549)
0928 16:50:45.227896 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:45.227993 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:45.230769 (+ 2776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:45.231881 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:45.231895 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:45.232783 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:45.233793 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:45.233826 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.235188 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.235194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:45.237271 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:45.237292 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.238369 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.238376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:45.238708 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:45.239537 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:45.239566 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:45.243905 (+ 4339us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":402,"threads_started":1}
W20260928 16:50:45.246006 18769 heartbeater.cc:644] Failed to heartbeat to 127.9.93.62:33515 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.93.62:33515: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Sep 28 16:50:45 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:45 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:45.256529 18771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:45.246696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.2:52313)
0928 16:50:45.246859 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:45.246879 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:45.247001 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:45.247396 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:45.247399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:45.247771 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:45.247966 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:45.247977 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.249541 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.249546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:45.250626 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:45.250644 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.251795 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.251802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:45.251979 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:45.252611 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:45.252633 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:45.256268 (+ 3635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 16:50:45.256887 18638 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:45.246823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:52313 (local address 127.9.93.62:33515)
0928 16:50:45.247011 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:45.247026 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:45.247050 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:50:45.247122 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:45.247127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:45.247228 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:45.247383 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:45.248132 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.249300 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.250808 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.251559 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.251708 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:45.256631 (+ 4923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.2:52313: BlockingRecv error: recv got EOF from 127.9.93.2:52313 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 16:50:45 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:45 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:45.266820 18771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:45.257637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.2:49419)
0928 16:50:45.257852 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:45.257874 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:45.258039 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:45.258391 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:45.258396 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:45.258755 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:45.258990 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:45.259012 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.260212 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.260217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:45.261231 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:45.261247 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.262014 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.262020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:45.262209 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:45.262831 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:45.262852 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:45.266413 (+ 3561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 16:50:45.266978 18638 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:45.257713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:49419 (local address 127.9.93.62:33515)
0928 16:50:45.257885 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:45.257891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:45.257950 (+ 59us) server_negotiation.cc:407] Connection header received
0928 16:50:45.258184 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:45.258190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:45.258259 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:45.258341 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:45.259162 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.260051 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.261389 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.261838 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.261957 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:45.266734 (+ 4777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.2:49419: BlockingRecv error: recv got EOF from 127.9.93.2:49419 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 16:50:45.267421 18769 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:50:45.617735 18772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:50:45.618104 18772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:50:45.618271 18772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:50:45.626693 18772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:50:45.626865 18772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:50:45.627054 18772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:50:45.638729 18772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:42831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/krb5kdc/oryx_127.9.93.3.keytab
--principal=oryx/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:43933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=41885
--tserver_master_addrs=127.9.93.62:33515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:50:45.640477 18772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:50:45.642239 18772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:50:45.658635 18777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:45.658478 18780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:50:45.658968 18778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:50:45.659869 18772 server_base.cc:1034] running on GCE node
I20260928 16:50:45.660743 18772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:50:45.662003 18772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:50:45.663291 18772 hybrid_clock.cc:648] HybridClock initialized: now 1790614245663207 us; error 67 us; skew 500 ppm
I20260928 16:50:45.663712 18772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:50:45 dist-test-slave-wzcb krb5kdc[17894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614245, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:50:45.667666 18772 init.cc:377] Logged in from keytab as oryx/127.9.93.3@KRBTEST.COM (short username oryx)
I20260928 16:50:45.669628 18772 webserver.cc:460] Webserver started at http://127.9.93.3:41885/ using document root <none> and password file <none>
I20260928 16:50:45.670471 18772 fs_manager.cc:359] Metadata directory not provided
I20260928 16:50:45.670640 18772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:50:45.677814 18772 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:50:45.681751 18787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:45.683887 18772 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:50:45.684121 18772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "609077d1f9194fb9aa6002c180b7c834"
format_stamp: "Formatted at 2026-09-28 16:50:42 on dist-test-slave-wzcb"
server_key: "7e5d0bfe6aab3f31537a5e5aab76d9dc"
server_key_iv: "46f9e96e2eb8a50b12392cc636484412"
server_key_version: "encryptionkey@0"
I20260928 16:50:45.685324 18772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:50:45.701694 18772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:50:45.707226 18772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:50:45.707653 18772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:50:45.708920 18772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:50:45.710722 18772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:50:45.710839 18772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:45.710999 18772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:50:45.711087 18772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:50:45.751160 18772 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:43933
I20260928 16:50:45.751209 18900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:43933 every 8 connection(s)
I20260928 16:50:45.753121 18772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:50:45.761087 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 18772
Sep 28 16:50:45 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:45 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 16:50:45.780848 18638 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:45.752057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:50547 (local address 127.9.93.62:33515)
0928 16:50:45.752306 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:45.752315 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:45.767439 (+ 15124us) server_negotiation.cc:407] Connection header received
0928 16:50:45.768042 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:45.768048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:45.768153 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:45.768258 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:45.769848 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.771135 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.773410 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.774218 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.774335 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:45.780529 (+ 6194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.1:50547: BlockingRecv error: recv got EOF from 127.9.93.1:50547 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 16:50:45.781667 18901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:45.751925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.1:50547)
0928 16:50:45.767256 (+ 15331us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:45.767286 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:45.767869 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:45.768492 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:45.768498 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:45.769289 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:45.769614 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:45.769631 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.771426 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.771434 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:45.773165 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:45.773187 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.774722 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.774729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:45.774945 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:45.775795 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:45.775823 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:45.781336 (+ 5513us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":15140,"thread_start_us":15562,"threads_started":1}
Sep 28 16:50:45 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:45 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 16:50:45.794180 18907 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:45.758716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:34729 (local address 127.9.93.62:33515)
0928 16:50:45.776203 (+ 17487us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:45.776210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:45.779374 (+ 3164us) server_negotiation.cc:407] Connection header received
0928 16:50:45.782242 (+ 2868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:45.782246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:45.782371 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:45.782496 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:45.784563 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.785489 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.787477 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.787929 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.788045 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:45.793918 (+ 5873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.3:34729: BlockingRecv error: recv got EOF from 127.9.93.3:34729 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":17304,"thread_start_us":18360,"threads_started":1}
I20260928 16:50:45.795166 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:45.757745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.3:34729)
0928 16:50:45.778105 (+ 20360us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:45.778190 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:45.781233 (+ 3043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:45.782869 (+ 1636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:45.782883 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:45.783611 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:45.784350 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:45.784374 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.785674 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.785678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:45.787274 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:45.787291 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.788207 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.788215 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:45.788474 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:45.789257 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:45.789315 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:45.793616 (+ 4301us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":19840,"thread_start_us":20089,"threads_started":1}
W20260928 16:50:45.796079 18902 heartbeater.cc:644] Failed to heartbeat to 127.9.93.62:33515 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.93.62:33515: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
I20260928 16:50:45.796388 18905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:45.763044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50658 (local address 127.9.93.62:33515)
0928 16:50:45.777359 (+ 14315us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:45.777365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:45.777390 (+ 25us) server_negotiation.cc:407] Connection header received
0928 16:50:45.777472 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:45.777479 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:45.777594 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:45.777743 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:45.782240 (+ 4497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.783376 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.784790 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.785320 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.785529 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:45.788514 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:50:45.788524 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:50:45.788543 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:50:45.788592 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:50:45.792234 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:45.793074 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:45.793083 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:45.793091 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:45.793295 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:50:45.793820 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:50:45.793826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:50:45.793829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:50:45.794634 (+ 805us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:50:45.794956 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0928 16:50:45.795538 (+ 582us) server_negotiation.cc:299] Negotiation successful
0928 16:50:45.795964 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":14195,"thread_start_us":12783,"threads_started":1}
Sep 28 16:50:45 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:45 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:45.805310 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:45.796745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.3:37571)
0928 16:50:45.796942 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:45.796963 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:45.797101 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:45.797601 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:45.797606 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:45.798033 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:45.798216 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:45.798225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.799381 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.799385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:45.800450 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:45.800466 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.801186 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.801192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:45.801364 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:45.801956 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:45.801976 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:45.804962 (+ 2986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 16:50:45.805595 18905 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:45.797096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:37571 (local address 127.9.93.62:33515)
0928 16:50:45.797335 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:45.797339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:45.797354 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:50:45.797397 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:45.797400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:45.797454 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:45.797529 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:45.798369 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.799191 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.800600 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.801032 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.801146 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:45.805309 (+ 4163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.3:37571: BlockingRecv error: recv got EOF from 127.9.93.3:37571 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
Sep 28 16:50:45 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:45 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:45.816174 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:45.806167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.3:39583)
0928 16:50:45.806355 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:45.806375 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:45.806506 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:45.806857 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:45.806862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:45.807404 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:45.807617 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:45.807627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.808769 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.808775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:45.810269 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:45.810287 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.811607 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.811620 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:45.811819 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:45.812528 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:45.812553 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:45.815875 (+ 3322us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 16:50:45.816638 18902 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:50:45.816927 18905 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:45.806342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:39583 (local address 127.9.93.62:33515)
0928 16:50:45.806574 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:45.806580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:45.806598 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:50:45.806656 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:45.806660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:45.806720 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:45.806800 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:45.807770 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.808591 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.810480 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:45.811185 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:45.811374 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:45.816167 (+ 4793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.3:39583: BlockingRecv error: recv got EOF from 127.9.93.3:39583 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 28 16:50:46 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:46 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:46.281659 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:46.268160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.2:44015)
0928 16:50:46.268709 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:46.268728 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:46.268873 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:46.269240 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:46.269243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:46.269724 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:46.269915 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:46.269926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:46.271157 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:46.271162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:46.272233 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:46.272249 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:46.273151 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:46.273156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:46.273299 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:46.277544 (+ 4245us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:46.277579 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:46.281288 (+ 3709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":262,"threads_started":1}
W20260928 16:50:46.281826 18905 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:46.268279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:44015 (local address 127.9.93.62:33515)
0928 16:50:46.268515 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:46.268520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:46.268850 (+ 330us) server_negotiation.cc:407] Connection header received
0928 16:50:46.269013 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:46.269018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:46.269105 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:46.269205 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:46.270098 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:46.270988 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:46.272449 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:46.272954 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:46.273071 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:46.281584 (+ 8513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.2:44015: BlockingRecv error: recv got EOF from 127.9.93.2:44015 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 16:50:46 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:46 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:46.792624 18909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:46.783200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.1:32823)
0928 16:50:46.783860 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:46.783878 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:46.784010 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:46.784366 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:46.784371 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:46.784881 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:46.785067 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:46.785078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:46.786187 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:46.786192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:46.787284 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:46.787344 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:46.788149 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:46.788154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:46.788300 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:46.788935 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:46.788954 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:46.792293 (+ 3339us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":386,"threads_started":1}
W20260928 16:50:46.792876 18910 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:46.783237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:32823 (local address 127.9.93.62:33515)
0928 16:50:46.783816 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:46.783821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:46.783954 (+ 133us) server_negotiation.cc:407] Connection header received
0928 16:50:46.784105 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:46.784109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:46.784183 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:46.784279 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:46.785225 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:46.786007 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:46.787496 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:46.787961 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:46.788067 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:46.792599 (+ 4532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.1:32823: BlockingRecv error: recv got EOF from 127.9.93.1:32823 (error 108)
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":295,"threads_started":1}
Sep 28 16:50:46 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:46 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:46.827672 18911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:46.817436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.3:45823)
0928 16:50:46.818089 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:46.818124 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:46.818270 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:46.818648 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:46.818652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:46.819338 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:46.819600 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:46.819613 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:46.820878 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:46.820884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:46.822042 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:46.822063 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:46.822970 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:46.822976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:46.823140 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:46.823829 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:46.823851 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:46.827386 (+ 3535us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":365,"threads_started":1}
W20260928 16:50:46.828022 18910 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:46.817553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:45823 (local address 127.9.93.62:33515)
0928 16:50:46.817772 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:46.817778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:46.818219 (+ 441us) server_negotiation.cc:407] Connection header received
0928 16:50:46.818379 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:46.818384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:46.818458 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:46.818595 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:46.819751 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:46.820690 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:46.822239 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:46.822764 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:46.822889 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:46.827745 (+ 4856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.3:45823: BlockingRecv error: recv got EOF from 127.9.93.3:45823 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 16:50:47 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:47 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:47.292992 18912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:47.282940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.2:36957)
0928 16:50:47.283684 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:47.283702 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:47.283851 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:47.284326 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:47.284332 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:47.284933 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:47.285143 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:47.285155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:47.286524 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:47.286530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:47.287562 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:47.287578 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:47.288401 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:47.288407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:47.288554 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:47.289202 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:47.289221 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:47.292628 (+ 3407us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":399,"threads_started":1}
W20260928 16:50:47.293213 18910 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:47.283078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:36957 (local address 127.9.93.62:33515)
0928 16:50:47.283263 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:47.283268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:47.283810 (+ 542us) server_negotiation.cc:407] Connection header received
0928 16:50:47.283981 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:47.283987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:47.284083 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:47.284213 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:47.285305 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:47.286345 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:47.287723 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:47.288183 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:47.288304 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:47.292925 (+ 4621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.2:36957: BlockingRecv error: recv got EOF from 127.9.93.2:36957 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 16:50:47 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:47 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:47.803275 18913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:47.793790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.1:40343)
0928 16:50:47.794436 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:47.794459 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:47.794601 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:47.794957 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:47.794961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:47.795504 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:47.795740 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:47.795753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:47.796967 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:47.796972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:47.798051 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:47.798068 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:47.798914 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:47.798920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:47.799089 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:47.799796 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:47.799817 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:47.802981 (+ 3164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":280,"threads_started":1}
W20260928 16:50:47.803670 18914 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:47.794056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:40343 (local address 127.9.93.62:33515)
0928 16:50:47.794553 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:47.794558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:47.794576 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:50:47.794731 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:47.794735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:47.794801 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:47.794892 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:47.795930 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:47.796797 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:47.798228 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:47.798739 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:47.798846 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:47.803352 (+ 4506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.1:40343: BlockingRecv error: recv got EOF from 127.9.93.1:40343 (error 108)
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":254,"threads_started":1}
Sep 28 16:50:47 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:47 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:47.838711 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:47.828876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.3:43939)
0928 16:50:47.829548 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:47.829571 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:47.829709 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:47.830121 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:47.830128 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:47.830633 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:47.830828 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:47.830839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:47.832028 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:47.832035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:47.833059 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:47.833077 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:47.833852 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:47.833859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:47.834041 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:47.834771 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:47.834796 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:47.838301 (+ 3505us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":371,"threads_started":1}
W20260928 16:50:47.838845 18914 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:47.828919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:43939 (local address 127.9.93.62:33515)
0928 16:50:47.829110 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:47.829114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:47.829662 (+ 548us) server_negotiation.cc:407] Connection header received
0928 16:50:47.829821 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:47.829825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:47.829893 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:47.830032 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:47.831010 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:47.831852 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:47.833239 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:47.833671 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:47.833795 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:47.838599 (+ 4804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.3:43939: BlockingRecv error: recv got EOF from 127.9.93.3:43939 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 16:50:48 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:48 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:48.303702 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:48.294252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.2:49925)
0928 16:50:48.294824 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:48.294842 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:48.295021 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:48.295453 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:48.295458 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:48.295889 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:48.296105 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:48.296117 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:48.297217 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:48.297221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:48.298198 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:48.298213 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:48.298943 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:48.298949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:48.299139 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:48.299873 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:48.299894 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:48.303373 (+ 3479us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":318,"threads_started":1}
W20260928 16:50:48.303805 18914 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:48.294354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:49925 (local address 127.9.93.62:33515)
0928 16:50:48.294554 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:48.294562 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:48.294946 (+ 384us) server_negotiation.cc:407] Connection header received
0928 16:50:48.295183 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:48.295187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:48.295272 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:48.295383 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:48.296272 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:48.297030 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:48.298355 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:48.298781 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:48.298894 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:48.303570 (+ 4676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.2:49925: BlockingRecv error: recv got EOF from 127.9.93.2:49925 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 16:50:48 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:48 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:48.814289 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:48.804381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.1:49371)
0928 16:50:48.804945 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:48.804964 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:48.805086 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:48.805557 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:48.805561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:48.806013 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:48.806197 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:48.806208 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:48.807606 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:48.807611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:48.808785 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:48.808802 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:48.809597 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:48.809603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:48.809782 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:48.810493 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:48.810516 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:48.814017 (+ 3501us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":283,"threads_started":1}
W20260928 16:50:48.814631 18918 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:48.804526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:49371 (local address 127.9.93.62:33515)
0928 16:50:48.805248 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:48.805253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:48.805269 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:50:48.805316 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:48.805320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:48.805394 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:48.805485 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:48.806363 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:48.807407 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:48.808962 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:48.809427 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:48.809529 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:48.814344 (+ 4815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.1:49371: BlockingRecv error: recv got EOF from 127.9.93.1:49371 (error 108)
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":467,"threads_started":1}
Sep 28 16:50:48 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:48 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:48.850201 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:48.840056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.3:55171)
0928 16:50:48.840696 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:48.840718 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:48.840954 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:48.841435 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:48.841441 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:48.841912 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:48.842179 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:48.842196 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:48.843561 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:48.843565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:48.844551 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:48.844567 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:48.845587 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:48.845592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:48.845759 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:48.846444 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:48.846467 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:48.849802 (+ 3335us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":312,"threads_started":1}
W20260928 16:50:48.850343 18918 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:48.840149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:55171 (local address 127.9.93.62:33515)
0928 16:50:48.840422 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:48.840428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:48.840930 (+ 502us) server_negotiation.cc:407] Connection header received
0928 16:50:48.841105 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:48.841111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:48.841205 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:48.841319 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:48.842343 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:48.843375 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:48.844831 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:48.845359 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:48.845518 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:48.850115 (+ 4597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.3:55171: BlockingRecv error: recv got EOF from 127.9.93.3:55171 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Sep 28 16:50:49 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:49 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:49.314824 18920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:49.305124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.2:41183)
0928 16:50:49.305834 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:49.305853 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:49.306058 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:49.306445 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:49.306450 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:49.306964 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:49.307251 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:49.307268 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:49.308425 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:49.308431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:49.309508 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:49.309524 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:49.310329 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:49.310335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:49.310554 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:49.311211 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:49.311233 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:49.314422 (+ 3189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":436,"threads_started":1}
W20260928 16:50:49.314997 18918 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:49.305265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:41183 (local address 127.9.93.62:33515)
0928 16:50:49.305520 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:49.305527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:49.305974 (+ 447us) server_negotiation.cc:407] Connection header received
0928 16:50:49.306201 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:49.306205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:49.306276 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:49.306367 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:49.307445 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:49.308266 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:49.309681 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:49.310162 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:49.310283 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:49.314711 (+ 4428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.2:41183: BlockingRecv error: recv got EOF from 127.9.93.2:41183 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 28 16:50:49 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:49 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:49.826411 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:49.815784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.1:35833)
0928 16:50:49.816443 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:49.816472 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:49.816673 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:49.817530 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:49.817536 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:49.818140 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:49.818413 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:49.818428 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:49.819610 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:49.819615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:49.820571 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:49.820585 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:49.821356 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:49.821362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:49.821572 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:49.822207 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:49.822227 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:49.826044 (+ 3817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":340,"threads_started":1}
W20260928 16:50:49.826634 18922 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:49.815878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:35833 (local address 127.9.93.62:33515)
0928 16:50:49.817158 (+ 1280us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:49.817164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:49.817182 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:50:49.817240 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:49.817244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:49.817321 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:49.817444 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:49.818574 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:49.819455 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:49.820726 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:49.821176 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:49.821295 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:49.826331 (+ 5036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.1:35833: BlockingRecv error: recv got EOF from 127.9.93.1:35833 (error 108)
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":600,"threads_started":1}
Sep 28 16:50:49 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:49 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:49.861826 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:49.851734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.3:46709)
0928 16:50:49.852289 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:49.852306 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:49.852456 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:49.852906 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:49.852910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:49.853352 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:49.853564 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:49.853576 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:49.854782 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:49.854788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:49.856234 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:49.856256 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:49.857163 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:49.857170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:49.857368 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:49.858031 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:49.858054 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:49.861538 (+ 3484us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":301,"threads_started":1}
W20260928 16:50:49.862319 18922 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:49.851814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:46709 (local address 127.9.93.62:33515)
0928 16:50:49.852054 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:49.852063 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:49.852395 (+ 332us) server_negotiation.cc:407] Connection header received
0928 16:50:49.852626 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:49.852632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:49.852720 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:49.852815 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:49.853755 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:49.854594 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:49.856424 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:49.856970 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:49.857109 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:49.861875 (+ 4766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.3:46709: BlockingRecv error: recv got EOF from 127.9.93.3:46709 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 16:50:50 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:50 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:50.326053 18924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:50.316399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.2:43609)
0928 16:50:50.317195 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:50.317220 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:50.317409 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:50.317795 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:50.317800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:50.318379 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:50.318600 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:50.318612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:50.319746 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:50.319751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:50.320752 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:50.320767 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:50.321561 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:50.321579 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:50.321789 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:50.322399 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:50.322417 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:50.325758 (+ 3341us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":487,"threads_started":1}
W20260928 16:50:50.326327 18922 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:50.316643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:43609 (local address 127.9.93.62:33515)
0928 16:50:50.316868 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:50.316875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:50.317337 (+ 462us) server_negotiation.cc:407] Connection header received
0928 16:50:50.317550 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:50.317554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:50.317648 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:50.317735 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:50.318753 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:50.319590 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:50.320910 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:50.321397 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:50.321518 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:50.326060 (+ 4542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.2:43609: BlockingRecv error: recv got EOF from 127.9.93.2:43609 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 16:50:50 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:50 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:50.839710 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:50.827922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.1:33935)
0928 16:50:50.828599 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:50.828623 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:50.828764 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:50.830329 (+ 1565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:50.830334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:50.830864 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:50.831072 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:50.831084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:50.832566 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:50.832572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:50.833895 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:50.833915 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:50.834817 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:50.834823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:50.834998 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:50.835744 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:50.835766 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:50.839352 (+ 3586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":352,"threads_started":1}
W20260928 16:50:50.839963 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:50.828093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:33935 (local address 127.9.93.62:33515)
0928 16:50:50.829892 (+ 1799us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:50.829899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:50.829925 (+ 26us) server_negotiation.cc:407] Connection header received
0928 16:50:50.830001 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:50.830005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:50.830091 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:50.830211 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:50.831338 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:50.832349 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:50.834092 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:50.834603 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:50.834719 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:50.839710 (+ 4991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.1:33935: BlockingRecv error: recv got EOF from 127.9.93.1:33935 (error 108)
Metrics: {"server-negotiator.queue_time_us":1659,"thread_start_us":1487,"threads_started":1}
Sep 28 16:50:50 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:50 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:50.873385 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:50.863046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.3:42123)
0928 16:50:50.863725 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:50.863745 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:50.863896 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:50.864392 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:50.864397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:50.864919 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:50.865126 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:50.865137 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:50.866434 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:50.866439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:50.867689 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:50.867712 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:50.868581 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:50.868588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:50.868768 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:50.869430 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:50.869451 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:50.873105 (+ 3654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":372,"threads_started":1}
W20260928 16:50:50.873576 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:50.863142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:42123 (local address 127.9.93.62:33515)
0928 16:50:50.863373 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:50.863378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:50.863875 (+ 497us) server_negotiation.cc:407] Connection header received
0928 16:50:50.864071 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:50.864075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:50.864153 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:50.864267 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:50.865323 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:50.866213 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:50.867875 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:50.868352 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:50.868465 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:50.873349 (+ 4884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.3:42123: BlockingRecv error: recv got EOF from 127.9.93.3:42123 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 16:50:51 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:51 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:51.343700 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:51.327359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.2:60189)
0928 16:50:51.328011 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:51.328030 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:51.328163 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:51.328581 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:51.328585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:51.329103 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:51.329319 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:51.329330 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:51.330431 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:51.330436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:51.331460 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:51.331477 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:51.332284 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:51.332289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:51.332426 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:51.339560 (+ 7134us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:51.339589 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:51.343413 (+ 3824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":389,"threads_started":1}
W20260928 16:50:51.343901 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:51.327415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:60189 (local address 127.9.93.62:33515)
0928 16:50:51.327605 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:51.327612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:51.328113 (+ 501us) server_negotiation.cc:407] Connection header received
0928 16:50:51.328260 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:51.328264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:51.328346 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:51.328437 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:51.329469 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:51.330250 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:51.331673 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:51.332134 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:51.332238 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:51.343649 (+ 11411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.2:60189: BlockingRecv error: recv got EOF from 127.9.93.2:60189 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 16:50:51 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:51 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:51.850220 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:51.840972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.1:45287)
0928 16:50:51.841628 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:51.841646 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:51.841790 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:51.842151 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:51.842157 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:51.842671 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:51.842887 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:51.842899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:51.844044 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:51.844049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:51.845051 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:51.845065 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:51.845859 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:51.845866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:51.846044 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:51.846619 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:51.846639 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:51.849829 (+ 3190us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":321,"threads_started":1}
W20260928 16:50:51.850406 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:51.841306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:45287 (local address 127.9.93.62:33515)
0928 16:50:51.841549 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:51.841556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:51.841721 (+ 165us) server_negotiation.cc:407] Connection header received
0928 16:50:51.841910 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:51.841916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:51.842003 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:51.842096 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:51.843038 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:51.843866 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:51.845201 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:51.845690 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:51.845807 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:51.850165 (+ 4358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.1:45287: BlockingRecv error: recv got EOF from 127.9.93.1:45287 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 28 16:50:51 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:51 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:51.884069 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:51.874654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.3:55607)
0928 16:50:51.875248 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:51.875265 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:51.875477 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:51.875893 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:51.875899 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:51.876363 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:51.876601 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:51.876613 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:51.877935 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:51.877940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:51.878924 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:51.878941 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:51.879769 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:51.879776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:51.879972 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:51.880593 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:51.880612 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:51.883785 (+ 3173us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":312,"threads_started":1}
W20260928 16:50:51.884299 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:51.874878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:55607 (local address 127.9.93.62:33515)
0928 16:50:51.875160 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:51.875168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:51.875365 (+ 197us) server_negotiation.cc:407] Connection header received
0928 16:50:51.875664 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:51.875668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:51.875746 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:51.875833 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:51.876769 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:51.877786 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:51.879081 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:51.879591 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:51.879709 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:51.884056 (+ 4347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.3:55607: BlockingRecv error: recv got EOF from 127.9.93.3:55607 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 28 16:50:52 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:52 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:52.354461 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:52.344848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.2:57921)
0928 16:50:52.345800 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:52.345819 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:52.345973 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:52.346384 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:52.346388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:52.346871 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:52.347085 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:52.347096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:52.348252 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:52.348257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:52.349264 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:52.349280 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:52.350016 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:52.350021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:52.350152 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:52.350811 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:52.350833 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:52.354097 (+ 3264us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":803,"thread_start_us":522,"threads_started":1}
W20260928 16:50:52.354642 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:52.344951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:57921 (local address 127.9.93.62:33515)
0928 16:50:52.345263 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:52.345272 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:52.345929 (+ 657us) server_negotiation.cc:407] Connection header received
0928 16:50:52.346140 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:52.346144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:52.346244 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:52.346338 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:52.347287 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:52.348081 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:52.349419 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:52.349864 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:52.349970 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:52.354385 (+ 4415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.2:57921: BlockingRecv error: recv got EOF from 127.9.93.2:57921 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
Sep 28 16:50:52 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:52 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:52.861075 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:52.851547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.1:54281)
0928 16:50:52.852138 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:52.852157 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:52.852297 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:52.852635 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:52.852638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:52.853087 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:52.853300 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:52.853312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:52.854579 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:52.854586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:52.855794 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:52.855813 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:52.856695 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:52.856700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:52.856835 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:52.857446 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:52.857468 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:52.860786 (+ 3318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":321,"threads_started":1}
W20260928 16:50:52.861268 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:52.851678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:54281 (local address 127.9.93.62:33515)
0928 16:50:52.851860 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:52.851865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:52.852233 (+ 368us) server_negotiation.cc:407] Connection header received
0928 16:50:52.852411 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:52.852415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:52.852486 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:52.852601 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:52.853459 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:52.854400 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:52.855939 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:52.856504 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:52.856612 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:52.861042 (+ 4430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.1:54281: BlockingRecv error: recv got EOF from 127.9.93.1:54281 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 16:50:52 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:52 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:52.894873 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:52.885216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.3:57727)
0928 16:50:52.885870 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:52.885890 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:52.886056 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:52.886405 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:52.886453 (+ 48us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:52.886921 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:52.887158 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:52.887170 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:52.888299 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:52.888304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:52.889341 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:52.889357 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:52.890247 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:52.890252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:52.890397 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:52.891071 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:52.891093 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:52.894570 (+ 3477us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":281,"threads_started":1}
W20260928 16:50:52.895025 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:52.885382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:57727 (local address 127.9.93.62:33515)
0928 16:50:52.885681 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:52.885687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:52.885993 (+ 306us) server_negotiation.cc:407] Connection header received
0928 16:50:52.886183 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:52.886187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:52.886260 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:52.886337 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:52.887296 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:52.888125 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:52.889494 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:52.890087 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:52.890193 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:52.894795 (+ 4602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.3:57727: BlockingRecv error: recv got EOF from 127.9.93.3:57727 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 28 16:50:53 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:53 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:53.366477 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:53.356130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.2:42153)
0928 16:50:53.356775 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:53.356794 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:53.356970 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:53.357397 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:53.357403 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:53.357937 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:53.358161 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:53.358173 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:53.359711 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:53.359716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:53.361018 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:53.361038 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:53.361984 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:53.361992 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:53.362155 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:53.362843 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:53.362865 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:53.366205 (+ 3340us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":306,"threads_started":1}
W20260928 16:50:53.366765 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:53.356293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:42153 (local address 127.9.93.62:33515)
0928 16:50:53.356568 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:53.356576 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:53.356894 (+ 318us) server_negotiation.cc:407] Connection header received
0928 16:50:53.357136 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:53.357141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:53.357226 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:53.357319 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:53.358347 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:53.359525 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:53.361186 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:53.361755 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:53.361877 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:53.366482 (+ 4605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.2:42153: BlockingRecv error: recv got EOF from 127.9.93.2:42153 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 28 16:50:53 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:53 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:53.872552 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:53.862294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.1:46159)
0928 16:50:53.862961 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:53.862980 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:53.863164 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:53.863587 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:53.863592 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:53.864117 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:53.864330 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:53.864341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:53.865536 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:53.865541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:53.866872 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:53.866897 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:53.867909 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:53.867923 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:53.868098 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:53.868770 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:53.868796 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:53.872264 (+ 3468us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":325,"threads_started":1}
W20260928 16:50:53.872831 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:53.862530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:46159 (local address 127.9.93.62:33515)
0928 16:50:53.862821 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:53.862828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:53.863083 (+ 255us) server_negotiation.cc:407] Connection header received
0928 16:50:53.863335 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:53.863339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:53.863413 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:53.863521 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:53.864519 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:53.865348 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:53.867054 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:53.867697 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:53.867833 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:53.872575 (+ 4742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.1:46159: BlockingRecv error: recv got EOF from 127.9.93.1:46159 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 28 16:50:53 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:53 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:53.906028 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:53.896033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.3:38659)
0928 16:50:53.896605 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:53.896623 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:53.896762 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:53.897249 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:53.897259 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:53.897783 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:53.897991 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:53.898002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:53.899171 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:53.899175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:53.900259 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:53.900277 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:53.901116 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:53.901123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:53.901282 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:53.901982 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:53.902005 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:53.905631 (+ 3626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":294,"threads_started":1}
W20260928 16:50:53.906307 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:53.896179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:38659 (local address 127.9.93.62:33515)
0928 16:50:53.896449 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:53.896456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:53.896715 (+ 259us) server_negotiation.cc:407] Connection header received
0928 16:50:53.896930 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:53.896936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:53.897049 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:53.897168 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:53.898180 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:53.898999 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:53.900410 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:53.900889 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:53.900994 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:53.905919 (+ 4925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.3:38659: BlockingRecv error: recv got EOF from 127.9.93.3:38659 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 28 16:50:54 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:54 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:54.377873 18937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:54.367795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.2:52199)
0928 16:50:54.368486 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:54.368505 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:54.368655 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:54.369012 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:54.369016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:54.369574 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:54.369781 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:54.369794 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:54.370987 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:54.370993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:54.372068 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:54.372084 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:54.372943 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:54.372949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:54.373120 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:54.373813 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:54.373836 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:54.377570 (+ 3734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":394,"threads_started":1}
W20260928 16:50:54.378028 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:54.367881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:52199 (local address 127.9.93.62:33515)
0928 16:50:54.368103 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:54.368108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:54.368601 (+ 493us) server_negotiation.cc:407] Connection header received
0928 16:50:54.368768 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:54.368772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:54.368847 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:54.368937 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:54.369976 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:54.370813 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:54.372227 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:54.372734 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:54.372843 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:54.377813 (+ 4970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.2:52199: BlockingRecv error: recv got EOF from 127.9.93.2:52199 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 16:50:54 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:54 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:54.883577 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:54.873813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.1:38553)
0928 16:50:54.874402 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:54.874422 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:54.874601 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:54.874966 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:54.874970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:54.875471 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:54.875698 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:54.875711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:54.877048 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:54.877054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:54.878415 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:54.878436 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:54.879252 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:54.879258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:54.879476 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:54.880095 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:54.880113 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:54.883268 (+ 3155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":317,"threads_started":1}
W20260928 16:50:54.883838 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:54.873995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:38553 (local address 127.9.93.62:33515)
0928 16:50:54.874310 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:54.874318 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:54.874525 (+ 207us) server_negotiation.cc:407] Connection header received
0928 16:50:54.874736 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:54.874741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:54.874834 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:54.874939 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:54.875882 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:54.876877 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:54.878608 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:54.879075 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:54.879191 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:54.883594 (+ 4403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.1:38553: BlockingRecv error: recv got EOF from 127.9.93.1:38553 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Sep 28 16:50:54 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:54 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:54.917486 18939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:54.907183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.3:57275)
0928 16:50:54.907886 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:54.907905 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:54.908059 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:54.908562 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:54.908567 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:54.909033 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:54.909265 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:54.909276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:54.910496 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:54.910501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:54.911675 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:54.911693 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:54.912693 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:54.912698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:54.912854 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:54.913476 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:54.913496 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:54.917036 (+ 3540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":389,"threads_started":1}
W20260928 16:50:54.917548 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:54.907275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:57275 (local address 127.9.93.62:33515)
0928 16:50:54.907544 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:54.907549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:54.908009 (+ 460us) server_negotiation.cc:407] Connection header received
0928 16:50:54.908230 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:54.908235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:54.908325 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:54.908472 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:54.909503 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:54.910314 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:54.911863 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:54.912494 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:54.912622 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:54.917308 (+ 4686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.3:57275: BlockingRecv error: recv got EOF from 127.9.93.3:57275 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 28 16:50:55 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:55 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:55.389099 18940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:55.379218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.2:58355)
0928 16:50:55.379873 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:55.379931 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:55.380117 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:55.380530 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:55.380535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:55.381072 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:55.381281 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:55.381292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:55.382510 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:55.382514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:55.383531 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:55.383546 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:55.384356 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:55.384362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:55.384561 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:55.385182 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:55.385206 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:55.388793 (+ 3587us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":387,"threads_started":1}
W20260928 16:50:55.389310 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:55.379410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:58355 (local address 127.9.93.62:33515)
0928 16:50:55.379732 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:55.379738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:55.380041 (+ 303us) server_negotiation.cc:407] Connection header received
0928 16:50:55.380278 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:55.380283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:55.380368 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:55.380469 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:55.381489 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:55.382340 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:55.383700 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:55.384179 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:55.384309 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:55.389046 (+ 4737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.2:58355: BlockingRecv error: recv got EOF from 127.9.93.2:58355 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Sep 28 16:50:55 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:55 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:55.893802 18941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:55.884629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.1:58113)
0928 16:50:55.885238 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:55.885256 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:55.885412 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:55.885769 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:55.885775 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:55.886264 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:55.886478 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:55.886499 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:55.887631 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:55.887636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:55.888593 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:55.888607 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:55.889373 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:55.889380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:55.889580 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:55.890293 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:55.890314 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:55.893529 (+ 3215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":341,"threads_started":1}
W20260928 16:50:55.894117 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:55.884860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:58113 (local address 127.9.93.62:33515)
0928 16:50:55.885062 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:55.885066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:55.885352 (+ 286us) server_negotiation.cc:407] Connection header received
0928 16:50:55.885536 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:55.885542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:55.885623 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:55.885709 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:55.886658 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:55.887470 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:55.888742 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:55.889185 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:55.889329 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:55.893784 (+ 4455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.1:58113: BlockingRecv error: recv got EOF from 127.9.93.1:58113 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 16:50:55 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:55 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:55.928229 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:55.918811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.3:36797)
0928 16:50:55.919439 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:55.919458 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:55.919603 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:55.919928 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:55.919931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:55.920364 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:55.920549 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:55.920560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:55.921736 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:55.921743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:55.923023 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:55.923043 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:55.923949 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:55.923956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:55.924145 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:55.924744 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:55.924764 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:55.927958 (+ 3194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":333,"threads_started":1}
W20260928 16:50:55.928449 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:55.918894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:36797 (local address 127.9.93.62:33515)
0928 16:50:55.919101 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:55.919106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:55.919535 (+ 429us) server_negotiation.cc:407] Connection header received
0928 16:50:55.919727 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:55.919731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:55.919805 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:55.919889 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:55.920694 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:55.921498 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:55.923203 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:55.923768 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:55.923888 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:55.928224 (+ 4336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.3:36797: BlockingRecv error: recv got EOF from 127.9.93.3:36797 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 28 16:50:56 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:56 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:56.404209 18943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:56.390473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.2:57597)
0928 16:50:56.391058 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:56.391078 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:56.391224 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:56.391671 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:56.391675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:56.392196 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:56.392438 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:56.392449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:56.393577 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:56.393582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:56.394622 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:56.394641 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:56.395542 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:56.395548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:56.395714 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:56.400182 (+ 4468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:56.400209 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:56.403813 (+ 3604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":289,"threads_started":1}
W20260928 16:50:56.404412 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:56.390573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:57597 (local address 127.9.93.62:33515)
0928 16:50:56.390775 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:56.390780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:56.391160 (+ 380us) server_negotiation.cc:407] Connection header received
0928 16:50:56.391354 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:56.391359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:56.391431 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:56.391523 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:56.392608 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:56.393401 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:56.394806 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:56.395283 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:56.395435 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:56.404113 (+ 8678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.2:57597: BlockingRecv error: recv got EOF from 127.9.93.2:57597 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 28 16:50:56 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:56 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:56.904700 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:56.894975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.1:45623)
0928 16:50:56.895755 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:56.895775 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:56.895918 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:56.896242 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:56.896246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:56.896721 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:56.896929 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:56.896940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:56.897990 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:56.897996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:56.899144 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:56.899166 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:56.900246 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:56.900253 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:56.900461 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:56.901179 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:56.901199 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:56.904393 (+ 3194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":319,"threads_started":1}
W20260928 16:50:56.904940 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:56.895127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:45623 (local address 127.9.93.62:33515)
0928 16:50:56.895475 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:56.895482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:56.895851 (+ 369us) server_negotiation.cc:407] Connection header received
0928 16:50:56.896021 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:56.896025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:56.896098 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:56.896205 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:56.897060 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:56.897854 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:56.899379 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:56.900064 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:56.900224 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:56.904665 (+ 4441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.1:45623: BlockingRecv error: recv got EOF from 127.9.93.1:45623 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 28 16:50:56 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:56 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:56.939023 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:56.929347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.3:42667)
0928 16:50:56.929900 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:56.929919 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:56.930063 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:56.930378 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:56.930381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:56.930854 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:56.931090 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:56.931101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:56.932323 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:56.932328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:56.933304 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:56.933318 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:56.934236 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:56.934241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:56.934404 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:56.935067 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:56.935087 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:56.938653 (+ 3566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":297,"threads_started":1}
W20260928 16:50:56.939177 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:56.929497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:42667 (local address 127.9.93.62:33515)
0928 16:50:56.929749 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:56.929755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:56.930009 (+ 254us) server_negotiation.cc:407] Connection header received
0928 16:50:56.930177 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:56.930181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:56.930251 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:56.930336 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:56.931270 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:56.932165 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:56.933509 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:56.934014 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:56.934128 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:56.938911 (+ 4783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.3:42667: BlockingRecv error: recv got EOF from 127.9.93.3:42667 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 28 16:50:57 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:57 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:57.415076 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:57.405563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.2:47393)
0928 16:50:57.406136 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:57.406156 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:57.406298 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:57.406632 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:57.406636 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:57.407159 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:57.407455 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:57.407473 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:57.408671 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:57.408676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:57.409666 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:57.409680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:57.410427 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:57.410431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:57.410599 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:57.411292 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:57.411341 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:57.414813 (+ 3472us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":296,"threads_started":1}
W20260928 16:50:57.415344 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:57.405623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:47393 (local address 127.9.93.62:33515)
0928 16:50:57.405877 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:57.405885 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:57.406244 (+ 359us) server_negotiation.cc:407] Connection header received
0928 16:50:57.406406 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:57.406411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:57.406499 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:57.406592 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:57.407599 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:57.408492 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:57.409824 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:57.410289 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:57.410390 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:57.415060 (+ 4670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.2:47393: BlockingRecv error: recv got EOF from 127.9.93.2:47393 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 28 16:50:57 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:57 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:57.915298 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:57.905908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.1:49385)
0928 16:50:57.906510 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:57.906529 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:57.906667 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:57.907133 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:57.907139 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:57.907690 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:57.907887 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:57.907897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:57.909072 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:57.909077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:57.910115 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:57.910130 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:57.911069 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:57.911075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:57.911231 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:57.911912 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:57.911932 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:57.914994 (+ 3062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":296,"threads_started":1}
W20260928 16:50:57.915797 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:57.905959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:49385 (local address 127.9.93.62:33515)
0928 16:50:57.906231 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:57.906239 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:57.906607 (+ 368us) server_negotiation.cc:407] Connection header received
0928 16:50:57.906826 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:57.906831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:57.906906 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:57.907047 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:57.908053 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:57.908899 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:57.910296 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:57.910855 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:57.910970 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:57.915279 (+ 4309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.1:49385: BlockingRecv error: recv got EOF from 127.9.93.1:49385 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 16:50:57 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:57 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:57.949864 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:57.940300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.3:54199)
0928 16:50:57.940968 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:57.940987 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:57.941174 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:57.941602 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:57.941607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:57.942147 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:57.942352 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:57.942367 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:57.943804 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:57.943809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:57.944833 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:57.944850 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:57.945699 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:57.945703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:57.945861 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:57.946522 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:57.946543 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:57.949588 (+ 3045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":365,"threads_started":1}
W20260928 16:50:57.950073 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:57.940350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:54199 (local address 127.9.93.62:33515)
0928 16:50:57.940551 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:57.940558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:57.941105 (+ 547us) server_negotiation.cc:407] Connection header received
0928 16:50:57.941319 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:57.941323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:57.941401 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:57.941516 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:57.942522 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:57.943548 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:57.944998 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:57.945554 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:57.945665 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:57.949837 (+ 4172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.3:54199: BlockingRecv error: recv got EOF from 127.9.93.3:54199 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 16:50:58 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:58 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:58.427009 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:58.416483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.2:54479)
0928 16:50:58.417251 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:58.417272 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:58.417476 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:58.417951 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:58.417958 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:58.418520 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:58.418718 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:58.418730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:58.419999 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:58.420004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:58.421185 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:58.421207 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:58.422229 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:58.422236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:58.422457 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:58.423190 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:58.423218 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:58.426691 (+ 3473us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":445,"threads_started":1}
W20260928 16:50:58.427351 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:58.416710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:54479 (local address 127.9.93.62:33515)
0928 16:50:58.417047 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:58.417055 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:58.417439 (+ 384us) server_negotiation.cc:407] Connection header received
0928 16:50:58.417649 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:58.417655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:58.417734 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:58.417832 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:58.418911 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:58.419797 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:58.421389 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:58.422024 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:58.422172 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:58.427030 (+ 4858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.2:54479: BlockingRecv error: recv got EOF from 127.9.93.2:54479 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 28 16:50:58 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:58 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:58.925989 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:58.916707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.1:40789)
0928 16:50:58.917310 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:58.917328 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:58.917458 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:58.917782 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:58.917786 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:58.918245 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:58.918458 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:58.918471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:58.919660 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:58.919666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:58.920691 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:58.920708 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:58.921479 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:58.921487 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:58.921667 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:58.922317 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:58.922336 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:58.925611 (+ 3275us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":334,"threads_started":1}
W20260928 16:50:58.926198 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:58.916831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:40789 (local address 127.9.93.62:33515)
0928 16:50:58.917020 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:58.917025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:58.917399 (+ 374us) server_negotiation.cc:407] Connection header received
0928 16:50:58.917556 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:58.917560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:58.917655 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:58.917745 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:58.918603 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:58.919459 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:58.920861 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:58.921308 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:58.921428 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:58.925935 (+ 4507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.1:40789: BlockingRecv error: recv got EOF from 127.9.93.1:40789 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 16:50:58 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:58 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:58.961053 18951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:58.951380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.3:35117)
0928 16:50:58.952014 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:58.952032 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:58.952213 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:58.952602 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:58.952609 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:58.953092 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:58.953279 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:58.953290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:58.954476 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:58.954481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:58.955541 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:58.955558 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:58.956441 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:58.956449 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:58.956625 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:58.957340 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:58.957362 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:58.960769 (+ 3407us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":424,"threads_started":1}
W20260928 16:50:58.961488 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:58.951536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:35117 (local address 127.9.93.62:33515)
0928 16:50:58.951808 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:58.951813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:58.952195 (+ 382us) server_negotiation.cc:407] Connection header received
0928 16:50:58.952362 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:58.952366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:58.952445 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:58.952539 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:58.953477 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:58.954294 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:58.955695 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:58.956221 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:58.956357 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:58.961222 (+ 4865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.3:35117: BlockingRecv error: recv got EOF from 127.9.93.3:35117 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
Sep 28 16:50:59 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:59 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:59.437357 18952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:59.428128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.2:54117)
0928 16:50:59.428758 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:59.428778 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:59.428934 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:59.429294 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:59.429299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:59.429721 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:59.429940 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:59.429949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:59.431043 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:59.431048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:59.432116 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:59.432131 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:59.432896 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:59.432903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:59.433094 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:59.433758 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:59.433779 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:59.437003 (+ 3224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":334,"threads_started":1}
W20260928 16:50:59.437582 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:59.428253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:54117 (local address 127.9.93.62:33515)
0928 16:50:59.428464 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:59.428471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:59.428873 (+ 402us) server_negotiation.cc:407] Connection header received
0928 16:50:59.429064 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:59.429069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:59.429138 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:59.429233 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:59.430106 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:59.430885 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:59.432276 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:59.432729 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:59.432851 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:59.437291 (+ 4440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.2:54117: BlockingRecv error: recv got EOF from 127.9.93.2:54117 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 16:50:59 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:59 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.1@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:59.937261 18953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:59.927768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.1:44149)
0928 16:50:59.928418 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:59.928441 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:59.928588 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:59.928990 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:59.928996 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:59.929478 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:59.929736 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:59.929749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:59.930913 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:59.930918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:59.931975 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:59.931992 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:59.932765 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:59.932770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:59.932905 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:59.933533 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:59.933553 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:59.936898 (+ 3345us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":333,"threads_started":1}
W20260928 16:50:59.937489 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:59.927827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:44149 (local address 127.9.93.62:33515)
0928 16:50:59.928038 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:59.928047 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:59.928530 (+ 483us) server_negotiation.cc:407] Connection header received
0928 16:50:59.928743 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:59.928747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:59.928830 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:59.928927 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:59.929944 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:59.930764 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:59.932129 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:59.932616 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:59.932727 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:59.937255 (+ 4528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.1:44149: BlockingRecv error: recv got EOF from 127.9.93.1:44149 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 16:50:59 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:50:59 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.3@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:50:59.972364 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:50:59.962258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.3:38393)
0928 16:50:59.962883 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0928 16:50:59.962903 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:50:59.963050 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:50:59.963431 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:50:59.963435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:50:59.963874 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:50:59.964109 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:59.964121 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:59.965411 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:59.965417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:59.966679 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:50:59.966701 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:50:59.967793 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:50:59.967799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:50:59.967975 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:59.968598 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:50:59.968638 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:50:59.971951 (+ 3313us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":368,"threads_started":1}
W20260928 16:50:59.972550 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:50:59.962396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:38393 (local address 127.9.93.62:33515)
0928 16:50:59.962588 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 16:50:59.962593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:50:59.962988 (+ 395us) server_negotiation.cc:407] Connection header received
0928 16:50:59.963165 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:50:59.963169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:50:59.963244 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:50:59.963388 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:50:59.964291 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:59.965219 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:59.966860 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:50:59.967557 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:50:59.967706 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:50:59.972272 (+ 4566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.3:38393: BlockingRecv error: recv got EOF from 127.9.93.3:38393 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 16:51:00 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:51:00 dist-test-slave-wzcb krb5kdc[17894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.93.2@KRBTEST.COM for oryx/127.9.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:51:00.448552 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:00.438873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33515 (local address 127.9.93.2:50149)
0928 16:51:00.439519 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0928 16:51:00.439539 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:51:00.439703 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:51:00.440053 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:51:00.440058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:51:00.440549 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:51:00.440776 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:51:00.440787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:51:00.441928 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:51:00.441932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:51:00.443018 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:51:00.443033 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:51:00.443895 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:51:00.443901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:51:00.444082 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:00.444689 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:51:00.444708 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:51:00.448164 (+ 3456us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.93.62:33515: Server oryx/127.9.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":305,"threads_started":1}
W20260928 16:51:00.448604 18926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:51:00.438958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:50149 (local address 127.9.93.62:33515)
0928 16:51:00.439206 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:00.439212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:00.439633 (+ 421us) server_negotiation.cc:407] Connection header received
0928 16:51:00.439837 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:00.439841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:00.439911 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:00.440007 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:00.440934 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:00.441782 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:00.443175 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:00.443709 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:00.443835 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:00.448415 (+ 4580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.93.2:50149: BlockingRecv error: recv got EOF from 127.9.93.2:50149 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:51:00.763206 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 18487
I20260928 16:51:00.780494 18957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:00.764628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46240 (local address 127.9.93.1:38551)
0928 16:51:00.765346 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:00.765355 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:00.765383 (+ 28us) server_negotiation.cc:407] Connection header received
0928 16:51:00.765451 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:00.765459 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:00.765708 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:00.765930 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:00.767727 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:00.769567 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:00.771144 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:00.772266 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:00.772434 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:00.775640 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:51:00.775657 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:51:00.775682 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:51:00.775748 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:51:00.779622 (+ 3874us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.93.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 16:51:00.780093 (+ 471us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46240: Request ticket server kudu/127.9.93.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":318,"threads_started":1}
W20260928 16:51:00.780593 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 18487: Not authorized: Client connection negotiation failed: client connection to 127.9.93.1:38551: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.93.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 16:51:00.780753 18957 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46240: Request ticket server kudu/127.9.93.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 16:51:00.780853 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 18487
I20260928 16:51:00.797351 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 18640
I20260928 16:51:00.813655 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:00.798586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42700 (local address 127.9.93.2:37639)
0928 16:51:00.799342 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:00.799354 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:00.799391 (+ 37us) server_negotiation.cc:407] Connection header received
0928 16:51:00.799467 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:00.799475 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:00.799731 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:00.799902 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:00.801857 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:00.803018 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:00.805165 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:00.805965 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:00.806104 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:00.809953 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:51:00.809967 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:51:00.809986 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:51:00.810052 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:51:00.813030 (+ 2978us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.93.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 16:51:00.813385 (+ 355us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42700: Request ticket server kudu/127.9.93.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":381,"threads_started":1}
W20260928 16:51:00.813773 18958 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42700: Request ticket server kudu/127.9.93.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 16:51:00.814097 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 18640: Not authorized: Client connection negotiation failed: client connection to 127.9.93.2:37639: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.93.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 16:51:00.814277 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 18640
I20260928 16:51:00.830224 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 18772
I20260928 16:51:00.846750 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:00.831454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55798 (local address 127.9.93.3:43933)
0928 16:51:00.832357 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:00.832365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:00.832394 (+ 29us) server_negotiation.cc:407] Connection header received
0928 16:51:00.832466 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:00.832475 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:00.832729 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:00.832896 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:00.834967 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:00.836238 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:00.838291 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:00.839007 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:00.839152 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:00.842863 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:51:00.842875 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:51:00.842893 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:51:00.842949 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:51:00.846080 (+ 3131us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.93.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 16:51:00.846441 (+ 361us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55798: Request ticket server kudu/127.9.93.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":513,"threads_started":1}
W20260928 16:51:00.846877 18959 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55798: Request ticket server kudu/127.9.93.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 16:51:00.847193 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 18772: Not authorized: Client connection negotiation failed: client connection to 127.9.93.3:43933: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.93.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 16:51:00.847558 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 18772
I20260928 16:51:00.862815 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 18418
I20260928 16:51:00.864703 18480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:51:00.994961 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 18418
2026-09-28T16:51:01Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (22124 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T16:51:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:51:01Z Disabled control of system clock
I20260928 16:51:01.081389 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:35269
--webserver_interface=127.9.93.62
--webserver_port=0
--builtin_ntp_servers=127.9.93.20:42865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:35269
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 16:51:01.448271 18964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:51:01.448606 18964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:51:01.448707 18964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:51:01.456300 18964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:51:01.456429 18964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:51:01.456470 18964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:51:01.456553 18964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:51:01.456621 18964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:51:01.466423 18964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:42865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:35269
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:35269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:51:01.468039 18964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:51:01.470059 18964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:51:01.483004 18969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:51:01.483096 18970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:51:01.483207 18964 server_base.cc:1034] running on GCE node
W20260928 16:51:01.483428 18972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:51:01.484532 18964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:51:01.486280 18964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:51:01.487500 18964 hybrid_clock.cc:648] HybridClock initialized: now 1790614261487436 us; error 73 us; skew 500 ppm
I20260928 16:51:01.487887 18964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:51:01.494161 18964 webserver.cc:460] Webserver started at http://127.9.93.62:46015/ using document root <none> and password file <none>
I20260928 16:51:01.495030 18964 fs_manager.cc:359] Metadata directory not provided
I20260928 16:51:01.495160 18964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:51:01.495558 18964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:51:01.498991 18964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "1ca9d014f78740a3b2e740eba4c170f1"
format_stamp: "Formatted at 2026-09-28 16:51:01 on dist-test-slave-wzcb"
server_key: "0e13cf08ded93d040baf68b1acc0f977"
server_key_iv: "e699282ae58a56cc0b16a1c42368ff8d"
server_key_version: "encryptionkey@0"
I20260928 16:51:01.499960 18964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "1ca9d014f78740a3b2e740eba4c170f1"
format_stamp: "Formatted at 2026-09-28 16:51:01 on dist-test-slave-wzcb"
server_key: "0e13cf08ded93d040baf68b1acc0f977"
server_key_iv: "e699282ae58a56cc0b16a1c42368ff8d"
server_key_version: "encryptionkey@0"
I20260928 16:51:01.506008 18964 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260928 16:51:01.510268 18978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:01.512046 18964 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 16:51:01.512264 18964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "1ca9d014f78740a3b2e740eba4c170f1"
format_stamp: "Formatted at 2026-09-28 16:51:01 on dist-test-slave-wzcb"
server_key: "0e13cf08ded93d040baf68b1acc0f977"
server_key_iv: "e699282ae58a56cc0b16a1c42368ff8d"
server_key_version: "encryptionkey@0"
I20260928 16:51:01.512506 18964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:51:01.536237 18964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:51:01.537235 18964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:51:01.537547 18964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:51:01.557158 18964 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:35269
I20260928 16:51:01.557199 19030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:35269 every 8 connection(s)
I20260928 16:51:01.558938 18964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:51:01.563555 19031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:51:01.569108 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 18964
I20260928 16:51:01.569298 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:51:01.569999 9588 external_mini_cluster.cc:1428] Setting key 2439e522f4f3172e2185429b86ead35d
I20260928 16:51:01.577390 19031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ca9d014f78740a3b2e740eba4c170f1: Bootstrap starting.
I20260928 16:51:01.581897 19031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ca9d014f78740a3b2e740eba4c170f1: Neither blocks nor log segments found. Creating new log.
I20260928 16:51:01.583348 19031 log.cc:824] T 00000000000000000000000000000000 P 1ca9d014f78740a3b2e740eba4c170f1: Log is configured to *not* fsync() on all Append() calls
I20260928 16:51:01.587845 19031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ca9d014f78740a3b2e740eba4c170f1: No bootstrap required, opened a new log
I20260928 16:51:01.588460 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:01.572531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42288 (local address 127.9.93.62:35269)
0928 16:51:01.573912 (+ 1381us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:01.573931 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:01.574004 (+ 73us) server_negotiation.cc:407] Connection header received
0928 16:51:01.576518 (+ 2514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:01.576553 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:01.577300 (+ 747us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:01.577845 (+ 545us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:01.578488 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:01.580616 (+ 2128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:01.582258 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:01.583139 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:01.583411 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:01.584879 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:51:01.584894 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:51:01.584918 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:51:01.585785 (+ 867us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:51:01.585906 (+ 121us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:51:01.585952 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:51:01.586094 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:01.586382 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 16:51:01.586801 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":418,"threads_started":1}
I20260928 16:51:01.596329 19031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ca9d014f78740a3b2e740eba4c170f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca9d014f78740a3b2e740eba4c170f1" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 35269 } }
I20260928 16:51:01.596815 19031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ca9d014f78740a3b2e740eba4c170f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:51:01.596903 19031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ca9d014f78740a3b2e740eba4c170f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ca9d014f78740a3b2e740eba4c170f1, State: Initialized, Role: FOLLOWER
I20260928 16:51:01.597540 19031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ca9d014f78740a3b2e740eba4c170f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca9d014f78740a3b2e740eba4c170f1" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 35269 } }
I20260928 16:51:01.597787 19031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ca9d014f78740a3b2e740eba4c170f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:51:01.597945 19031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ca9d014f78740a3b2e740eba4c170f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:51:01.598155 19031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ca9d014f78740a3b2e740eba4c170f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:51:01.600708 19031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ca9d014f78740a3b2e740eba4c170f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca9d014f78740a3b2e740eba4c170f1" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 35269 } }
I20260928 16:51:01.601342 19031 leader_election.cc:302] T 00000000000000000000000000000000 P 1ca9d014f78740a3b2e740eba4c170f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ca9d014f78740a3b2e740eba4c170f1; no voters:
I20260928 16:51:01.602058 19031 leader_election.cc:288] T 00000000000000000000000000000000 P 1ca9d014f78740a3b2e740eba4c170f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:51:01.602259 19036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ca9d014f78740a3b2e740eba4c170f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:51:01.603035 19036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ca9d014f78740a3b2e740eba4c170f1 [term 1 LEADER]: Becoming Leader. State: Replica: 1ca9d014f78740a3b2e740eba4c170f1, State: Running, Role: LEADER
I20260928 16:51:01.603904 19036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ca9d014f78740a3b2e740eba4c170f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca9d014f78740a3b2e740eba4c170f1" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 35269 } }
I20260928 16:51:01.604729 19031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ca9d014f78740a3b2e740eba4c170f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:51:01.607352 19037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ca9d014f78740a3b2e740eba4c170f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ca9d014f78740a3b2e740eba4c170f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca9d014f78740a3b2e740eba4c170f1" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 35269 } } }
I20260928 16:51:01.607523 19038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ca9d014f78740a3b2e740eba4c170f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ca9d014f78740a3b2e740eba4c170f1. Latest consensus state: current_term: 1 leader_uuid: "1ca9d014f78740a3b2e740eba4c170f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca9d014f78740a3b2e740eba4c170f1" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 35269 } } }
I20260928 16:51:01.607856 19037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ca9d014f78740a3b2e740eba4c170f1 [sys.catalog]: This master's current role is: LEADER
I20260928 16:51:01.607906 19038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ca9d014f78740a3b2e740eba4c170f1 [sys.catalog]: This master's current role is: LEADER
I20260928 16:51:01.609688 19044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:51:01.615465 19044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:51:01.626330 19044 catalog_manager.cc:1347] Generated new cluster ID: 56649d31d782469dadbea35e5629adef
I20260928 16:51:01.626489 19044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:51:01.648594 19044 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:51:01.650244 19044 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:51:01.656767 19044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ca9d014f78740a3b2e740eba4c170f1: Generated new TSK 0
I20260928 16:51:01.657945 19044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:51:01.665100 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:35269
--builtin_ntp_servers=127.9.93.20:42865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 16:51:02.057076 19055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:51:02.057428 19055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:51:02.057566 19055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:51:02.065981 19055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:51:02.066125 19055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:51:02.066284 19055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:51:02.076905 19055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:42865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:35269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:51:02.078430 19055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:51:02.080614 19055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:51:02.096526 19063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:51:02.096809 19055 server_base.cc:1034] running on GCE node
W20260928 16:51:02.096592 19060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:51:02.097178 19061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:51:02.098021 19055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:51:02.099233 19055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:51:02.100548 19055 hybrid_clock.cc:648] HybridClock initialized: now 1790614262100466 us; error 45 us; skew 500 ppm
I20260928 16:51:02.100970 19055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:51:02.103971 19055 webserver.cc:460] Webserver started at http://127.9.93.1:38615/ using document root <none> and password file <none>
I20260928 16:51:02.104893 19055 fs_manager.cc:359] Metadata directory not provided
I20260928 16:51:02.105044 19055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:51:02.105409 19055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:51:02.109292 19055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "94549156734f4abea2c82a059cdf913b"
format_stamp: "Formatted at 2026-09-28 16:51:02 on dist-test-slave-wzcb"
server_key: "7c7a051cb78fa7b3d9a4c96bdd62fbd8"
server_key_iv: "41e1f6b1da257a52ff575caf51d6a8b1"
server_key_version: "encryptionkey@0"
I20260928 16:51:02.110368 19055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "94549156734f4abea2c82a059cdf913b"
format_stamp: "Formatted at 2026-09-28 16:51:02 on dist-test-slave-wzcb"
server_key: "7c7a051cb78fa7b3d9a4c96bdd62fbd8"
server_key_iv: "41e1f6b1da257a52ff575caf51d6a8b1"
server_key_version: "encryptionkey@0"
I20260928 16:51:02.117034 19055 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 16:51:02.121625 19069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:02.123430 19055 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:51:02.123701 19055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "94549156734f4abea2c82a059cdf913b"
format_stamp: "Formatted at 2026-09-28 16:51:02 on dist-test-slave-wzcb"
server_key: "7c7a051cb78fa7b3d9a4c96bdd62fbd8"
server_key_iv: "41e1f6b1da257a52ff575caf51d6a8b1"
server_key_version: "encryptionkey@0"
I20260928 16:51:02.123966 19055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:51:02.137913 19055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:51:02.138989 19055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:51:02.139336 19055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:51:02.140563 19055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:51:02.142426 19055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:51:02.142549 19055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:02.142668 19055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:51:02.142746 19055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:02.177402 19055 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:35659
I20260928 16:51:02.177444 19182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:35659 every 8 connection(s)
I20260928 16:51:02.178965 19055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:51:02.183935 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 19055
I20260928 16:51:02.184178 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:51:02.185088 9588 external_mini_cluster.cc:1428] Setting key 56502f369da58d99f38ee341f748d1f2
I20260928 16:51:02.188848 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:35269
--builtin_ntp_servers=127.9.93.20:42865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 16:51:02.196190 19185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:02.182000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:35591 (local address 127.9.93.62:35269)
0928 16:51:02.182532 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:02.182540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:02.184551 (+ 2011us) server_negotiation.cc:407] Connection header received
0928 16:51:02.187159 (+ 2608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:02.187165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:02.187245 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:02.187418 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:02.189213 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:02.190018 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:02.192645 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:02.193141 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:02.193277 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:02.195198 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:51:02.195204 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:51:02.195208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:51:02.195234 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:51:02.195283 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:51:02.195297 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:51:02.195422 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:02.195747 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 16:51:02.195862 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":251,"threads_started":1}
I20260928 16:51:02.197989 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:02.182764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:35269 (local address 127.9.93.1:35591)
0928 16:51:02.184282 (+ 1518us) negotiation.cc:107] Waiting for socket to connect
0928 16:51:02.184343 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 16:51:02.186806 (+ 2463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:51:02.187693 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:51:02.187706 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:51:02.187871 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:51:02.188903 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:51:02.188940 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:51:02.190222 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:51:02.190229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:51:02.192387 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:51:02.192412 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:51:02.193439 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:51:02.193446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:51:02.193838 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:02.194710 (+ 872us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:51:02.194713 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:51:02.194790 (+ 77us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:51:02.194920 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:51:02.195434 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:51:02.195446 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:51:02.195455 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 16:51:02.195691 (+ 236us) client_negotiation.cc:240] Negotiation successful
0928 16:51:02.196222 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"thread_start_us":455,"threads_started":1}
I20260928 16:51:02.200026 19183 heartbeater.cc:347] Connected to a master server at 127.9.93.62:35269
I20260928 16:51:02.200474 19183 heartbeater.cc:464] Registering TS with master...
I20260928 16:51:02.201442 19183 heartbeater.cc:511] Master 127.9.93.62:35269 requested a full tablet report, sending...
I20260928 16:51:02.204370 18996 ts_manager.cc:194] Registered new tserver with Master: 94549156734f4abea2c82a059cdf913b (127.9.93.1:35659)
I20260928 16:51:02.206676 18996 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.93.1:35591
W20260928 16:51:02.568943 19187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:51:02.569310 19187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:51:02.569453 19187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:51:02.577782 19187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:51:02.577942 19187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:51:02.578118 19187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:51:02.588874 19187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:42865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:35269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:51:02.590471 19187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:51:02.592684 19187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:51:02.608222 19193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:51:02.608248 19192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:51:02.608314 19195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:51:02.609560 19187 server_base.cc:1034] running on GCE node
I20260928 16:51:02.610373 19187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:51:02.611563 19187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:51:02.612792 19187 hybrid_clock.cc:648] HybridClock initialized: now 1790614262612710 us; error 68 us; skew 500 ppm
I20260928 16:51:02.613142 19187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:51:02.615860 19187 webserver.cc:460] Webserver started at http://127.9.93.2:41619/ using document root <none> and password file <none>
I20260928 16:51:02.616737 19187 fs_manager.cc:359] Metadata directory not provided
I20260928 16:51:02.616843 19187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:51:02.617209 19187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:51:02.620726 19187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "82a9a70b0aa04a9e938c8d43469ec724"
format_stamp: "Formatted at 2026-09-28 16:51:02 on dist-test-slave-wzcb"
server_key: "166a761d282e4268f2925f105584268f"
server_key_iv: "79d772f2931bbb283cea49fb758542b4"
server_key_version: "encryptionkey@0"
I20260928 16:51:02.621703 19187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "82a9a70b0aa04a9e938c8d43469ec724"
format_stamp: "Formatted at 2026-09-28 16:51:02 on dist-test-slave-wzcb"
server_key: "166a761d282e4268f2925f105584268f"
server_key_iv: "79d772f2931bbb283cea49fb758542b4"
server_key_version: "encryptionkey@0"
I20260928 16:51:02.628270 19187 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 16:51:02.632977 19201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:02.634729 19187 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:51:02.634976 19187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "82a9a70b0aa04a9e938c8d43469ec724"
format_stamp: "Formatted at 2026-09-28 16:51:02 on dist-test-slave-wzcb"
server_key: "166a761d282e4268f2925f105584268f"
server_key_iv: "79d772f2931bbb283cea49fb758542b4"
server_key_version: "encryptionkey@0"
I20260928 16:51:02.635262 19187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:51:02.648828 19187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:51:02.649991 19187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:51:02.650360 19187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:51:02.651728 19187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:51:02.653687 19187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:51:02.653810 19187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:02.653949 19187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:51:02.654044 19187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:02.692008 19187 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:37851
I20260928 16:51:02.692047 19315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:37851 every 8 connection(s)
I20260928 16:51:02.693630 19187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:51:02.701264 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 19187
I20260928 16:51:02.701443 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:51:02.701982 9588 external_mini_cluster.cc:1428] Setting key 3c405c3702046842d8b8753a7fae0ca5
I20260928 16:51:02.705374 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:35269
--builtin_ntp_servers=127.9.93.20:42865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 16:51:02.709764 19318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:02.696703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:49087 (local address 127.9.93.62:35269)
0928 16:51:02.697226 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:02.697231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:02.698818 (+ 1587us) server_negotiation.cc:407] Connection header received
0928 16:51:02.701129 (+ 2311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:02.701135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:02.701241 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:02.701346 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:02.702933 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:02.703903 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:02.706192 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:02.706680 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:02.706829 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:02.708710 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:51:02.708716 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:51:02.708720 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:51:02.708744 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:51:02.708795 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:51:02.708808 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:51:02.708896 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:02.709230 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 16:51:02.709387 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":291,"threads_started":1}
I20260928 16:51:02.711203 19319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:02.697380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:35269 (local address 127.9.93.2:49087)
0928 16:51:02.698594 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0928 16:51:02.698656 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 16:51:02.700697 (+ 2041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:51:02.701725 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:51:02.701737 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:51:02.701876 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:51:02.702685 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:51:02.702711 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:51:02.704118 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:51:02.704123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:51:02.705963 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:51:02.705990 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:51:02.707016 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:51:02.707027 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:51:02.707376 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:02.708258 (+ 882us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:51:02.708261 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:51:02.708328 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:51:02.708444 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:51:02.708944 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:51:02.708960 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:51:02.708968 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:51:02.709177 (+ 209us) client_negotiation.cc:240] Negotiation successful
0928 16:51:02.709766 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":357,"threads_started":1}
I20260928 16:51:02.713303 19316 heartbeater.cc:347] Connected to a master server at 127.9.93.62:35269
I20260928 16:51:02.713714 19316 heartbeater.cc:464] Registering TS with master...
I20260928 16:51:02.714699 19316 heartbeater.cc:511] Master 127.9.93.62:35269 requested a full tablet report, sending...
I20260928 16:51:02.716777 18996 ts_manager.cc:194] Registered new tserver with Master: 82a9a70b0aa04a9e938c8d43469ec724 (127.9.93.2:37851)
I20260928 16:51:02.717777 18996 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.93.2:49087
W20260928 16:51:03.086223 19320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:51:03.086570 19320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:51:03.086697 19320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:51:03.094928 19320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:51:03.095064 19320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:51:03.095230 19320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:51:03.105739 19320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:42865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:35269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:51:03.107359 19320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:51:03.109596 19320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:51:03.125119 19325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:51:03.125149 19326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:51:03.125178 19328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:51:03.125967 19320 server_base.cc:1034] running on GCE node
I20260928 16:51:03.126739 19320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:51:03.128116 19320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:51:03.129372 19320 hybrid_clock.cc:648] HybridClock initialized: now 1790614263129301 us; error 57 us; skew 500 ppm
I20260928 16:51:03.129802 19320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:51:03.132666 19320 webserver.cc:460] Webserver started at http://127.9.93.3:44915/ using document root <none> and password file <none>
I20260928 16:51:03.133519 19320 fs_manager.cc:359] Metadata directory not provided
I20260928 16:51:03.133651 19320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:51:03.134055 19320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:51:03.137701 19320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "7d0f129b9444437db5d5a0f4a424a321"
format_stamp: "Formatted at 2026-09-28 16:51:03 on dist-test-slave-wzcb"
server_key: "12c2ecdb037e0415b3208ba7baf715fb"
server_key_iv: "92edc29831b1afa07b3d8d336e1de566"
server_key_version: "encryptionkey@0"
I20260928 16:51:03.138691 19320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "7d0f129b9444437db5d5a0f4a424a321"
format_stamp: "Formatted at 2026-09-28 16:51:03 on dist-test-slave-wzcb"
server_key: "12c2ecdb037e0415b3208ba7baf715fb"
server_key_iv: "92edc29831b1afa07b3d8d336e1de566"
server_key_version: "encryptionkey@0"
I20260928 16:51:03.145244 19320 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 16:51:03.149868 19334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:03.151726 19320 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 16:51:03.151944 19320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "7d0f129b9444437db5d5a0f4a424a321"
format_stamp: "Formatted at 2026-09-28 16:51:03 on dist-test-slave-wzcb"
server_key: "12c2ecdb037e0415b3208ba7baf715fb"
server_key_iv: "92edc29831b1afa07b3d8d336e1de566"
server_key_version: "encryptionkey@0"
I20260928 16:51:03.152220 19320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:51:03.176307 19320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:51:03.177410 19320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:51:03.177739 19320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:51:03.178951 19320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:51:03.180997 19320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:51:03.181111 19320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:03.181239 19320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:51:03.181321 19320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:03.211141 19183 heartbeater.cc:503] Master 127.9.93.62:35269 was elected leader, sending a full tablet report...
I20260928 16:51:03.224673 19320 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:45187
I20260928 16:51:03.224745 19448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:45187 every 8 connection(s)
I20260928 16:51:03.226229 19320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:51:03.236405 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 19320
I20260928 16:51:03.236557 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:51:03.237035 9588 external_mini_cluster.cc:1428] Setting key 38e8c6f129542e3f990aa18d90dd3fd1
I20260928 16:51:03.241585 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:03.229305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:38341 (local address 127.9.93.62:35269)
0928 16:51:03.229785 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:03.229790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:03.231384 (+ 1594us) server_negotiation.cc:407] Connection header received
0928 16:51:03.233499 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:03.233506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:03.233608 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:03.233712 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:03.235039 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:03.235886 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:03.237860 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:03.238439 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:03.238569 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:03.240520 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:51:03.240526 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:51:03.240530 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:51:03.240552 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:51:03.240606 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:51:03.240620 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:51:03.240713 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:03.241003 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 16:51:03.241133 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":233,"threads_started":1}
I20260928 16:51:03.242836 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:03.230066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:35269 (local address 127.9.93.3:38341)
0928 16:51:03.231178 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0928 16:51:03.231230 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 16:51:03.233154 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:51:03.234008 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:51:03.234019 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:51:03.234144 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:51:03.234836 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:51:03.234856 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:51:03.236076 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:51:03.236081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:51:03.237654 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:51:03.237675 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:51:03.238753 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:51:03.238763 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:51:03.239097 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:03.240049 (+ 952us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:51:03.240052 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:51:03.240106 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:51:03.240220 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:51:03.240735 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:51:03.240752 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:51:03.240763 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 16:51:03.240981 (+ 218us) client_negotiation.cc:240] Negotiation successful
0928 16:51:03.241522 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":332,"threads_started":1}
I20260928 16:51:03.244498 19449 heartbeater.cc:347] Connected to a master server at 127.9.93.62:35269
I20260928 16:51:03.244853 19449 heartbeater.cc:464] Registering TS with master...
I20260928 16:51:03.245776 19449 heartbeater.cc:511] Master 127.9.93.62:35269 requested a full tablet report, sending...
I20260928 16:51:03.247440 18996 ts_manager.cc:194] Registered new tserver with Master: 7d0f129b9444437db5d5a0f4a424a321 (127.9.93.3:45187)
I20260928 16:51:03.248306 18996 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.93.3:38341
I20260928 16:51:03.253485 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 16:51:03.260752 19451 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:51:03.258750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55128 (local address 127.9.93.62:35269)
0928 16:51:03.258958 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:03.258964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:03.259674 (+ 710us) server_negotiation.cc:407] Connection header received
0928 16:51:03.259867 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:03.259871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:03.259948 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:03.260037 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:03.260476 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55128: BlockingRecv error: recv got EOF from 127.0.0.1:55128 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 16:51:03.261807 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 19055
I20260928 16:51:03.271152 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:03.262994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35840 (local address 127.9.93.1:35659)
0928 16:51:03.263783 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:03.263791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:03.263822 (+ 31us) server_negotiation.cc:407] Connection header received
0928 16:51:03.263890 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:03.263898 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:03.264159 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:03.264314 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:03.264734 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:03.265891 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:03.267286 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:03.268020 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:03.268139 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:03.269375 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:51:03.269387 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:51:03.269411 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:51:03.270138 (+ 727us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:51:03.270202 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:51:03.270227 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:51:03.270329 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:03.270561 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 16:51:03.270760 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":470,"threads_started":1}
I20260928 16:51:03.272636 19178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:51:03.361858 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 19055
I20260928 16:51:03.378019 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 19187
I20260928 16:51:03.387025 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:03.379123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44154 (local address 127.9.93.2:37851)
0928 16:51:03.379757 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:03.379766 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:03.379799 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:51:03.379870 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:03.379878 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:03.380149 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:03.380294 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:03.380711 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:03.381817 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:03.383223 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:03.384025 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:03.384177 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:03.385396 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:51:03.385404 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:51:03.385421 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:51:03.386148 (+ 727us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:51:03.386205 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:51:03.386228 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:51:03.386329 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:03.386535 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 16:51:03.386683 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":322,"threads_started":1}
I20260928 16:51:03.388590 19311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:51:03.475564 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 19187
I20260928 16:51:03.490942 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 19320
I20260928 16:51:03.500092 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:03.492136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52004 (local address 127.9.93.3:45187)
0928 16:51:03.492863 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:03.492871 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:03.492906 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:51:03.492979 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:03.492985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:03.493261 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:03.493407 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:03.493950 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:03.495082 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:03.496540 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:03.497218 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:03.497334 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:03.498473 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:51:03.498484 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:51:03.498501 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:51:03.499220 (+ 719us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:51:03.499281 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:51:03.499340 (+ 59us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:51:03.499441 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:03.499582 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 16:51:03.499734 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":383,"threads_started":1}
I20260928 16:51:03.501626 19444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:51:03.589704 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 19320
I20260928 16:51:03.604040 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 18964
I20260928 16:51:03.605477 19026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:51:03.714814 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 18964
2026-09-28T16:51:03Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2712 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T16:51:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:51:03Z Disabled control of system clock
I20260928 16:51:03.792403 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:40511
--webserver_interface=127.9.93.62
--webserver_port=0
--builtin_ntp_servers=127.9.93.20:38073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:40511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 16:51:04.158399 19469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:51:04.158756 19469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:51:04.158854 19469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:51:04.166584 19469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:51:04.166723 19469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:51:04.166761 19469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:51:04.166819 19469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:51:04.166862 19469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:51:04.176822 19469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:38073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:40511
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:40511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:51:04.178347 19469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:51:04.180451 19469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:51:04.193751 19474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:51:04.193780 19477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:51:04.193765 19475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:51:04.194468 19469 server_base.cc:1034] running on GCE node
I20260928 16:51:04.195538 19469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:51:04.197412 19469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:51:04.198678 19469 hybrid_clock.cc:648] HybridClock initialized: now 1790614264198630 us; error 56 us; skew 500 ppm
I20260928 16:51:04.199115 19469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:51:04.201917 19469 webserver.cc:460] Webserver started at http://127.9.93.62:33697/ using document root <none> and password file <none>
I20260928 16:51:04.202785 19469 fs_manager.cc:359] Metadata directory not provided
I20260928 16:51:04.202895 19469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:51:04.203265 19469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:51:04.208768 19469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "9e44f6d262d04dcba94084816f375fc9"
format_stamp: "Formatted at 2026-09-28 16:51:04 on dist-test-slave-wzcb"
server_key: "8120563639e339bb6ff192d8e9cb9b8d"
server_key_iv: "24d4ed068ee5b70912a27307e3e9e079"
server_key_version: "encryptionkey@0"
I20260928 16:51:04.209748 19469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "9e44f6d262d04dcba94084816f375fc9"
format_stamp: "Formatted at 2026-09-28 16:51:04 on dist-test-slave-wzcb"
server_key: "8120563639e339bb6ff192d8e9cb9b8d"
server_key_iv: "24d4ed068ee5b70912a27307e3e9e079"
server_key_version: "encryptionkey@0"
I20260928 16:51:04.216331 19469 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260928 16:51:04.220752 19483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:04.222574 19469 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260928 16:51:04.222795 19469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "9e44f6d262d04dcba94084816f375fc9"
format_stamp: "Formatted at 2026-09-28 16:51:04 on dist-test-slave-wzcb"
server_key: "8120563639e339bb6ff192d8e9cb9b8d"
server_key_iv: "24d4ed068ee5b70912a27307e3e9e079"
server_key_version: "encryptionkey@0"
I20260928 16:51:04.223048 19469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:51:04.229650 19469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:51:04.230655 19469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:51:04.230974 19469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:51:04.249027 19469 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:40511
I20260928 16:51:04.249047 19534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:40511 every 8 connection(s)
I20260928 16:51:04.250599 19469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:51:04.255077 19535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:51:04.259384 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 19469
I20260928 16:51:04.259624 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:51:04.260269 9588 external_mini_cluster.cc:1428] Setting key ab0a7c1c13c9139145dbb8f2c3e1b1a7
I20260928 16:51:04.267884 19535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e44f6d262d04dcba94084816f375fc9: Bootstrap starting.
I20260928 16:51:04.270227 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:04.262573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.9.93.62:40511)
0928 16:51:04.263763 (+ 1190us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:04.263776 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:04.263826 (+ 50us) server_negotiation.cc:407] Connection header received
0928 16:51:04.266073 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:04.266120 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:04.266653 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:04.267092 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:04.268338 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:51:04.268344 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:51:04.268360 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:51:04.268378 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:51:04.268450 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:51:04.268478 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:51:04.268706 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:04.268855 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 16:51:04.269149 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":396,"threads_started":1}
I20260928 16:51:04.271795 19535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e44f6d262d04dcba94084816f375fc9: Neither blocks nor log segments found. Creating new log.
I20260928 16:51:04.273473 19535 log.cc:824] T 00000000000000000000000000000000 P 9e44f6d262d04dcba94084816f375fc9: Log is configured to *not* fsync() on all Append() calls
I20260928 16:51:04.276782 19535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e44f6d262d04dcba94084816f375fc9: No bootstrap required, opened a new log
I20260928 16:51:04.285171 19535 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e44f6d262d04dcba94084816f375fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e44f6d262d04dcba94084816f375fc9" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 40511 } }
I20260928 16:51:04.285643 19535 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e44f6d262d04dcba94084816f375fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:51:04.285745 19535 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e44f6d262d04dcba94084816f375fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e44f6d262d04dcba94084816f375fc9, State: Initialized, Role: FOLLOWER
I20260928 16:51:04.286484 19535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e44f6d262d04dcba94084816f375fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e44f6d262d04dcba94084816f375fc9" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 40511 } }
I20260928 16:51:04.286774 19535 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e44f6d262d04dcba94084816f375fc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:51:04.286921 19535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e44f6d262d04dcba94084816f375fc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:51:04.287174 19535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e44f6d262d04dcba94084816f375fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:51:04.289628 19535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e44f6d262d04dcba94084816f375fc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e44f6d262d04dcba94084816f375fc9" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 40511 } }
I20260928 16:51:04.290270 19535 leader_election.cc:302] T 00000000000000000000000000000000 P 9e44f6d262d04dcba94084816f375fc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e44f6d262d04dcba94084816f375fc9; no voters:
I20260928 16:51:04.290988 19535 leader_election.cc:288] T 00000000000000000000000000000000 P 9e44f6d262d04dcba94084816f375fc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:51:04.291136 19540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e44f6d262d04dcba94084816f375fc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:51:04.291908 19540 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e44f6d262d04dcba94084816f375fc9 [term 1 LEADER]: Becoming Leader. State: Replica: 9e44f6d262d04dcba94084816f375fc9, State: Running, Role: LEADER
I20260928 16:51:04.292507 19540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e44f6d262d04dcba94084816f375fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e44f6d262d04dcba94084816f375fc9" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 40511 } }
I20260928 16:51:04.293413 19535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e44f6d262d04dcba94084816f375fc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:51:04.295611 19542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e44f6d262d04dcba94084816f375fc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e44f6d262d04dcba94084816f375fc9. Latest consensus state: current_term: 1 leader_uuid: "9e44f6d262d04dcba94084816f375fc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e44f6d262d04dcba94084816f375fc9" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 40511 } } }
I20260928 16:51:04.295933 19542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e44f6d262d04dcba94084816f375fc9 [sys.catalog]: This master's current role is: LEADER
I20260928 16:51:04.296347 19541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e44f6d262d04dcba94084816f375fc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e44f6d262d04dcba94084816f375fc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e44f6d262d04dcba94084816f375fc9" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 40511 } } }
I20260928 16:51:04.296686 19541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e44f6d262d04dcba94084816f375fc9 [sys.catalog]: This master's current role is: LEADER
I20260928 16:51:04.296921 19546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:51:04.303148 19546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:51:04.315619 19546 catalog_manager.cc:1347] Generated new cluster ID: e82d8251a432409181d351f35cc3f09a
I20260928 16:51:04.315763 19546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:51:04.336434 19546 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:51:04.336993 19546 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:51:04.344517 19546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e44f6d262d04dcba94084816f375fc9: Generated new TSK 0
I20260928 16:51:04.345547 19546 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:51:04.355707 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:40511
--builtin_ntp_servers=127.9.93.20:38073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 16:51:04.741638 19559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:51:04.742007 19559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:51:04.742164 19559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:51:04.750501 19559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:51:04.750662 19559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:51:04.750828 19559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:51:04.761304 19559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:38073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:40511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:51:04.762921 19559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:51:04.765280 19559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:51:04.780720 19565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:51:04.780668 19567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:51:04.780743 19564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:51:04.785243 19559 server_base.cc:1034] running on GCE node
I20260928 16:51:04.786067 19559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:51:04.787379 19559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:51:04.788625 19559 hybrid_clock.cc:648] HybridClock initialized: now 1790614264788559 us; error 52 us; skew 500 ppm
I20260928 16:51:04.789062 19559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:51:04.792021 19559 webserver.cc:460] Webserver started at http://127.9.93.1:32797/ using document root <none> and password file <none>
I20260928 16:51:04.793026 19559 fs_manager.cc:359] Metadata directory not provided
I20260928 16:51:04.793201 19559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:51:04.793669 19559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:51:04.797377 19559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "aaf2e387e2ca49e2b380e7d1bb1746ce"
format_stamp: "Formatted at 2026-09-28 16:51:04 on dist-test-slave-wzcb"
server_key: "6f7a23b62aebccf2f9c3c0a1eaf3ad68"
server_key_iv: "ecf0551b4a3472cb2e76874b8be0f396"
server_key_version: "encryptionkey@0"
I20260928 16:51:04.798383 19559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "aaf2e387e2ca49e2b380e7d1bb1746ce"
format_stamp: "Formatted at 2026-09-28 16:51:04 on dist-test-slave-wzcb"
server_key: "6f7a23b62aebccf2f9c3c0a1eaf3ad68"
server_key_iv: "ecf0551b4a3472cb2e76874b8be0f396"
server_key_version: "encryptionkey@0"
I20260928 16:51:04.805310 19559 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260928 16:51:04.809873 19573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:04.811825 19559 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:51:04.812036 19559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "aaf2e387e2ca49e2b380e7d1bb1746ce"
format_stamp: "Formatted at 2026-09-28 16:51:04 on dist-test-slave-wzcb"
server_key: "6f7a23b62aebccf2f9c3c0a1eaf3ad68"
server_key_iv: "ecf0551b4a3472cb2e76874b8be0f396"
server_key_version: "encryptionkey@0"
I20260928 16:51:04.812296 19559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:51:04.819396 19559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:51:04.820427 19559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:51:04.820745 19559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:51:04.821907 19559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:51:04.823820 19559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:51:04.823938 19559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:04.824072 19559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:51:04.824148 19559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:04.860250 19559 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:45949
I20260928 16:51:04.860311 19686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:45949 every 8 connection(s)
I20260928 16:51:04.861780 19559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:51:04.866843 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 19559
I20260928 16:51:04.867110 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:51:04.867952 9588 external_mini_cluster.cc:1428] Setting key 4550099c00c1e6d8d3e9ea8bc0d98742
I20260928 16:51:04.871570 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:40511
--builtin_ntp_servers=127.9.93.20:38073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 16:51:04.872159 19689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:04.864971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:42701 (local address 127.9.93.62:40511)
0928 16:51:04.865561 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:04.865569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:04.867170 (+ 1601us) server_negotiation.cc:407] Connection header received
0928 16:51:04.869335 (+ 2165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:04.869340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:04.869401 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:04.869506 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:04.871148 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:51:04.871155 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:51:04.871160 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:51:04.871188 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:51:04.871258 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:51:04.871281 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:51:04.871488 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:04.871667 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 16:51:04.871751 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":245,"threads_started":1}
I20260928 16:51:04.873215 19690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:04.865780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:40511 (local address 127.9.93.1:42701)
0928 16:51:04.866934 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0928 16:51:04.867007 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 16:51:04.868970 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:51:04.869848 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:51:04.869862 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:51:04.870026 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:51:04.870763 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:51:04.870766 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:51:04.870814 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:51:04.870927 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:51:04.871426 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:51:04.871435 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:51:04.871441 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:51:04.871711 (+ 270us) client_negotiation.cc:240] Negotiation successful
0928 16:51:04.872059 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":402,"threads_started":1}
I20260928 16:51:04.875200 19687 heartbeater.cc:347] Connected to a master server at 127.9.93.62:40511
I20260928 16:51:04.875617 19687 heartbeater.cc:464] Registering TS with master...
I20260928 16:51:04.876386 19687 heartbeater.cc:511] Master 127.9.93.62:40511 requested a full tablet report, sending...
I20260928 16:51:04.879526 19500 ts_manager.cc:194] Registered new tserver with Master: aaf2e387e2ca49e2b380e7d1bb1746ce (127.9.93.1:45949)
W20260928 16:51:05.262558 19691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:51:05.262928 19691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:51:05.263067 19691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:51:05.271451 19691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:51:05.271610 19691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:51:05.271795 19691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:51:05.282238 19691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:38073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:40511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:51:05.283934 19691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:51:05.286185 19691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:51:05.302248 19697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:51:05.302248 19696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:51:05.302261 19699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:51:05.303905 19691 server_base.cc:1034] running on GCE node
I20260928 16:51:05.304716 19691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:51:05.305979 19691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:51:05.307250 19691 hybrid_clock.cc:648] HybridClock initialized: now 1790614265307178 us; error 57 us; skew 500 ppm
I20260928 16:51:05.307709 19691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:51:05.310698 19691 webserver.cc:460] Webserver started at http://127.9.93.2:42203/ using document root <none> and password file <none>
I20260928 16:51:05.311714 19691 fs_manager.cc:359] Metadata directory not provided
I20260928 16:51:05.311863 19691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:51:05.312251 19691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:51:05.315812 19691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "1df6292bc0fc4b4c9fc9dc4d97f83fbc"
format_stamp: "Formatted at 2026-09-28 16:51:05 on dist-test-slave-wzcb"
server_key: "e97da53c3a4ad158116cce86ec5423fe"
server_key_iv: "578d010f0f61365499c378be63099cef"
server_key_version: "encryptionkey@0"
I20260928 16:51:05.316779 19691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "1df6292bc0fc4b4c9fc9dc4d97f83fbc"
format_stamp: "Formatted at 2026-09-28 16:51:05 on dist-test-slave-wzcb"
server_key: "e97da53c3a4ad158116cce86ec5423fe"
server_key_iv: "578d010f0f61365499c378be63099cef"
server_key_version: "encryptionkey@0"
I20260928 16:51:05.323184 19691 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 16:51:05.327863 19705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:05.329660 19691 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260928 16:51:05.329893 19691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "1df6292bc0fc4b4c9fc9dc4d97f83fbc"
format_stamp: "Formatted at 2026-09-28 16:51:05 on dist-test-slave-wzcb"
server_key: "e97da53c3a4ad158116cce86ec5423fe"
server_key_iv: "578d010f0f61365499c378be63099cef"
server_key_version: "encryptionkey@0"
I20260928 16:51:05.330175 19691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:51:05.337077 19691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:51:05.338085 19691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:51:05.338380 19691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:51:05.339547 19691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:51:05.341290 19691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:51:05.341400 19691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:05.341523 19691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:51:05.341598 19691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:05.376720 19691 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:42683
I20260928 16:51:05.376765 19818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:42683 every 8 connection(s)
I20260928 16:51:05.378278 19691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:51:05.382309 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 19691
I20260928 16:51:05.382555 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:51:05.383183 9588 external_mini_cluster.cc:1428] Setting key c3578f161060fb723b46e4acc67e09d4
I20260928 16:51:05.386575 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:40511
--builtin_ntp_servers=127.9.93.20:38073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 16:51:05.388725 19821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:05.381323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:44025 (local address 127.9.93.62:40511)
0928 16:51:05.381852 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:05.381858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:05.383670 (+ 1812us) server_negotiation.cc:407] Connection header received
0928 16:51:05.385864 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:05.385871 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:05.385960 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:05.386061 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:05.387853 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:51:05.387859 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:51:05.387863 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:51:05.387885 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:51:05.387949 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:51:05.387963 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:51:05.388104 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:05.388306 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 16:51:05.388387 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":274,"threads_started":1}
I20260928 16:51:05.389989 19822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:05.382119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:40511 (local address 127.9.93.2:44025)
0928 16:51:05.383472 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0928 16:51:05.383519 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:51:05.385537 (+ 2018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:51:05.386383 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:51:05.386394 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:51:05.386529 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:51:05.387433 (+ 904us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:51:05.387436 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:51:05.387502 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:51:05.387633 (+ 131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:51:05.388066 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:51:05.388080 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:51:05.388089 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 16:51:05.388412 (+ 323us) client_negotiation.cc:240] Negotiation successful
0928 16:51:05.388787 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":441,"threads_started":1}
I20260928 16:51:05.391810 19819 heartbeater.cc:347] Connected to a master server at 127.9.93.62:40511
I20260928 16:51:05.392187 19819 heartbeater.cc:464] Registering TS with master...
I20260928 16:51:05.392971 19819 heartbeater.cc:511] Master 127.9.93.62:40511 requested a full tablet report, sending...
I20260928 16:51:05.394934 19500 ts_manager.cc:194] Registered new tserver with Master: 1df6292bc0fc4b4c9fc9dc4d97f83fbc (127.9.93.2:42683)
W20260928 16:51:05.765101 19823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:51:05.765441 19823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:51:05.765553 19823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:51:05.773540 19823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:51:05.773671 19823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:51:05.773799 19823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:51:05.783874 19823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:38073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:40511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:51:05.785403 19823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:51:05.787503 19823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:51:05.802749 19828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:51:05.802688 19831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:51:05.802700 19829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:51:05.803578 19823 server_base.cc:1034] running on GCE node
I20260928 16:51:05.804368 19823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:51:05.805536 19823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:51:05.806775 19823 hybrid_clock.cc:648] HybridClock initialized: now 1790614265806715 us; error 52 us; skew 500 ppm
I20260928 16:51:05.807183 19823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:51:05.809999 19823 webserver.cc:460] Webserver started at http://127.9.93.3:37155/ using document root <none> and password file <none>
I20260928 16:51:05.810845 19823 fs_manager.cc:359] Metadata directory not provided
I20260928 16:51:05.810976 19823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:51:05.811391 19823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:51:05.814872 19823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "ab301ffeda064abeb4507dfaef5b0797"
format_stamp: "Formatted at 2026-09-28 16:51:05 on dist-test-slave-wzcb"
server_key: "afa69b86d1fd9489c889fe23e6790032"
server_key_iv: "0cf1869b49f3b413d90c3c00ddee9555"
server_key_version: "encryptionkey@0"
I20260928 16:51:05.815913 19823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "ab301ffeda064abeb4507dfaef5b0797"
format_stamp: "Formatted at 2026-09-28 16:51:05 on dist-test-slave-wzcb"
server_key: "afa69b86d1fd9489c889fe23e6790032"
server_key_iv: "0cf1869b49f3b413d90c3c00ddee9555"
server_key_version: "encryptionkey@0"
I20260928 16:51:05.822711 19823 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 16:51:05.827526 19837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:05.829387 19823 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:51:05.829604 19823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "ab301ffeda064abeb4507dfaef5b0797"
format_stamp: "Formatted at 2026-09-28 16:51:05 on dist-test-slave-wzcb"
server_key: "afa69b86d1fd9489c889fe23e6790032"
server_key_iv: "0cf1869b49f3b413d90c3c00ddee9555"
server_key_version: "encryptionkey@0"
I20260928 16:51:05.829874 19823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:51:05.837029 19823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:51:05.838120 19823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:51:05.838435 19823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:51:05.839653 19823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:51:05.841450 19823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:51:05.841550 19823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:05.841683 19823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:51:05.841758 19823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:05.877431 19823 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:44785
I20260928 16:51:05.877513 19950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:44785 every 8 connection(s)
I20260928 16:51:05.879030 19823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:51:05.883522 19687 heartbeater.cc:503] Master 127.9.93.62:40511 was elected leader, sending a full tablet report...
I20260928 16:51:05.885946 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 19823
I20260928 16:51:05.886220 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:51:05.886793 9588 external_mini_cluster.cc:1428] Setting key 858cb1acfbd7bea3e2a3d409cc532a18
I20260928 16:51:05.891044 19821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:05.882618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:38697 (local address 127.9.93.62:40511)
0928 16:51:05.882822 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:05.882829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:05.885468 (+ 2639us) server_negotiation.cc:407] Connection header received
0928 16:51:05.888192 (+ 2724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:05.888198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:05.888277 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:05.888381 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:05.890171 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:51:05.890177 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:51:05.890182 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:51:05.890207 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:51:05.890252 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:51:05.890269 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:51:05.890400 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:05.890678 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 16:51:05.890755 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 16:51:05.892382 19953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:05.883480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:40511 (local address 127.9.93.3:38697)
0928 16:51:05.885217 (+ 1737us) negotiation.cc:107] Waiting for socket to connect
0928 16:51:05.885279 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 16:51:05.887791 (+ 2512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:51:05.888754 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:51:05.888768 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:51:05.888947 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:51:05.889775 (+ 828us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:51:05.889778 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:51:05.889848 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:51:05.889973 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:51:05.890388 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:51:05.890398 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:51:05.890405 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 16:51:05.890771 (+ 366us) client_negotiation.cc:240] Negotiation successful
0928 16:51:05.891139 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"thread_start_us":717,"threads_started":1}
I20260928 16:51:05.894104 19951 heartbeater.cc:347] Connected to a master server at 127.9.93.62:40511
I20260928 16:51:05.894552 19951 heartbeater.cc:464] Registering TS with master...
I20260928 16:51:05.895414 19951 heartbeater.cc:511] Master 127.9.93.62:40511 requested a full tablet report, sending...
I20260928 16:51:05.897303 19500 ts_manager.cc:194] Registered new tserver with Master: ab301ffeda064abeb4507dfaef5b0797 (127.9.93.3:44785)
I20260928 16:51:05.903867 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 16:51:05.911062 19821 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:51:05.909207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36630 (local address 127.9.93.62:40511)
0928 16:51:05.909497 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:05.909505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:05.910012 (+ 507us) server_negotiation.cc:407] Connection header received
0928 16:51:05.910224 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:05.910228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:05.910293 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:05.910404 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:05.910818 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36630: BlockingRecv error: recv got EOF from 127.0.0.1:36630 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
I20260928 16:51:05.911752 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 19559
I20260928 16:51:05.916733 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:05.913441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36214 (local address 127.9.93.1:45949)
0928 16:51:05.914073 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:05.914081 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:05.914110 (+ 29us) server_negotiation.cc:407] Connection header received
0928 16:51:05.914178 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:05.914184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:05.914433 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:05.914616 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:05.915879 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:51:05.915886 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:51:05.915908 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:51:05.915922 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:51:05.915973 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:51:05.915988 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:51:05.916168 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:05.916242 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 16:51:05.916359 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":304,"threads_started":1}
I20260928 16:51:05.918248 19682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:51:06.008430 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 19559
I20260928 16:51:06.025235 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 19691
I20260928 16:51:06.029728 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:06.026348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45484 (local address 127.9.93.2:42683)
0928 16:51:06.026958 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:06.026975 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:06.027008 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:51:06.027089 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:06.027097 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:06.027436 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:06.027630 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:06.028822 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:51:06.028830 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:51:06.028843 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:51:06.028859 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:51:06.028919 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:51:06.028941 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:51:06.029098 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:06.029251 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 16:51:06.029404 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":249,"threads_started":1}
I20260928 16:51:06.031294 19814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:51:06.122314 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 19691
I20260928 16:51:06.137373 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 19823
I20260928 16:51:06.142289 19964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:06.138638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42566 (local address 127.9.93.3:44785)
0928 16:51:06.139461 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:06.139469 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:06.139506 (+ 37us) server_negotiation.cc:407] Connection header received
0928 16:51:06.139592 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:06.139600 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:06.139864 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:06.140048 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:06.141408 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:51:06.141415 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:51:06.141430 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:51:06.141451 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:51:06.141498 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:51:06.141513 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:51:06.141712 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:06.141802 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 16:51:06.141928 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":450,"threads_started":1}
I20260928 16:51:06.143910 19946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:51:06.231714 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 19823
I20260928 16:51:06.248198 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 19469
I20260928 16:51:06.249804 19530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:51:06.358150 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 19469
2026-09-28T16:51:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2643 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:51:06 dist-test-slave-wzcb krb5kdc[19969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:51:06 dist-test-slave-wzcb krb5kdc[19969](info): set up 2 sockets
Sep 28 16:51:06 dist-test-slave-wzcb krb5kdc[19969](info): commencing operation
krb5kdc: starting...
W20260928 16:51:07.402773 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:51:07 dist-test-slave-wzcb krb5kdc[19969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:51:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:51:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:51:07.676117 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:33955
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:32787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:33955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:51:08.057686 19982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:51:08.058044 19982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:51:08.058172 19982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:51:08.066330 19982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:51:08.066478 19982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:51:08.066525 19982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:51:08.066594 19982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:51:08.066660 19982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:51:08.078220 19982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:32787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:33955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:33955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:51:08.079873 19982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:51:08.081542 19982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:51:08.095111 19987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:51:08.095033 19988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:51:08.095080 19990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:51:08.096318 19982 server_base.cc:1034] running on GCE node
I20260928 16:51:08.097297 19982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:51:08.099056 19982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:51:08.100332 19982 hybrid_clock.cc:648] HybridClock initialized: now 1790614268100255 us; error 66 us; skew 500 ppm
I20260928 16:51:08.100818 19982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:51:08 dist-test-slave-wzcb krb5kdc[19969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614268, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:51:08.104698 19982 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:51:08.106566 19982 webserver.cc:460] Webserver started at http://127.9.93.62:37859/ using document root <none> and password file <none>
I20260928 16:51:08.107471 19982 fs_manager.cc:359] Metadata directory not provided
I20260928 16:51:08.107625 19982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:51:08.107954 19982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:51:08.111513 19982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "068026e9db2c411a82af5bcf5f653597"
format_stamp: "Formatted at 2026-09-28 16:51:08 on dist-test-slave-wzcb"
server_key: "4d9dcbc9226a683273f80f03d8b5b4f9"
server_key_iv: "3773000e565da10fe9e38d67d5ed2954"
server_key_version: "encryptionkey@0"
I20260928 16:51:08.112428 19982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "068026e9db2c411a82af5bcf5f653597"
format_stamp: "Formatted at 2026-09-28 16:51:08 on dist-test-slave-wzcb"
server_key: "4d9dcbc9226a683273f80f03d8b5b4f9"
server_key_iv: "3773000e565da10fe9e38d67d5ed2954"
server_key_version: "encryptionkey@0"
I20260928 16:51:08.118979 19982 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 16:51:08.123637 19997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:08.125556 19982 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:51:08.125792 19982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "068026e9db2c411a82af5bcf5f653597"
format_stamp: "Formatted at 2026-09-28 16:51:08 on dist-test-slave-wzcb"
server_key: "4d9dcbc9226a683273f80f03d8b5b4f9"
server_key_iv: "3773000e565da10fe9e38d67d5ed2954"
server_key_version: "encryptionkey@0"
I20260928 16:51:08.126029 19982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:51:08.160923 19982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:51:08.166550 19982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:51:08.166935 19982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:51:08.188074 19982 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:33955
I20260928 16:51:08.188122 20048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:33955 every 8 connection(s)
I20260928 16:51:08.189754 19982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:51:08.194339 20049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:51:08.195403 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 19982
I20260928 16:51:08.195649 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:51:08.196512 9588 external_mini_cluster.cc:1428] Setting key 67b7e1e30840421859d22529f29f9ed3
I20260928 16:51:08.210933 20049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 068026e9db2c411a82af5bcf5f653597: Bootstrap starting.
Sep 28 16:51:08 dist-test-slave-wzcb krb5kdc[19969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:51:08.215365 20049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 068026e9db2c411a82af5bcf5f653597: Neither blocks nor log segments found. Creating new log.
I20260928 16:51:08.216841 20049 log.cc:824] T 00000000000000000000000000000000 P 068026e9db2c411a82af5bcf5f653597: Log is configured to *not* fsync() on all Append() calls
I20260928 16:51:08.220845 20049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 068026e9db2c411a82af5bcf5f653597: No bootstrap required, opened a new log
I20260928 16:51:08.223440 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:08.199000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39556 (local address 127.9.93.62:33955)
0928 16:51:08.199964 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:08.199982 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:08.200036 (+ 54us) server_negotiation.cc:407] Connection header received
0928 16:51:08.201874 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:08.201905 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:08.202432 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:08.202866 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:08.204906 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:08.206374 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:08.208436 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:08.209282 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:08.209441 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:08.214677 (+ 5236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:51:08.214689 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:51:08.214708 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:51:08.214764 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:51:08.218634 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:51:08.219721 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:51:08.219730 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:51:08.219738 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:51:08.219920 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:51:08.220393 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:51:08.220396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:51:08.220398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:51:08.221013 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:51:08.221290 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:08.221771 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 16:51:08.222199 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":435,"threads_started":1}
I20260928 16:51:08.230191 20049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 068026e9db2c411a82af5bcf5f653597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "068026e9db2c411a82af5bcf5f653597" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33955 } }
I20260928 16:51:08.230677 20049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 068026e9db2c411a82af5bcf5f653597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:51:08.230777 20049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 068026e9db2c411a82af5bcf5f653597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 068026e9db2c411a82af5bcf5f653597, State: Initialized, Role: FOLLOWER
I20260928 16:51:08.231563 20049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 068026e9db2c411a82af5bcf5f653597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "068026e9db2c411a82af5bcf5f653597" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33955 } }
I20260928 16:51:08.231822 20049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 068026e9db2c411a82af5bcf5f653597 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:51:08.231962 20049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 068026e9db2c411a82af5bcf5f653597 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:51:08.232136 20049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 068026e9db2c411a82af5bcf5f653597 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:51:08.234700 20049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 068026e9db2c411a82af5bcf5f653597 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "068026e9db2c411a82af5bcf5f653597" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33955 } }
I20260928 16:51:08.235257 20049 leader_election.cc:302] T 00000000000000000000000000000000 P 068026e9db2c411a82af5bcf5f653597 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 068026e9db2c411a82af5bcf5f653597; no voters:
I20260928 16:51:08.235958 20049 leader_election.cc:288] T 00000000000000000000000000000000 P 068026e9db2c411a82af5bcf5f653597 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:51:08.236105 20054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 068026e9db2c411a82af5bcf5f653597 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:51:08.236770 20054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 068026e9db2c411a82af5bcf5f653597 [term 1 LEADER]: Becoming Leader. State: Replica: 068026e9db2c411a82af5bcf5f653597, State: Running, Role: LEADER
I20260928 16:51:08.237342 20054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 068026e9db2c411a82af5bcf5f653597 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "068026e9db2c411a82af5bcf5f653597" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33955 } }
I20260928 16:51:08.238142 20049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 068026e9db2c411a82af5bcf5f653597 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:51:08.240649 20056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 068026e9db2c411a82af5bcf5f653597 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 068026e9db2c411a82af5bcf5f653597. Latest consensus state: current_term: 1 leader_uuid: "068026e9db2c411a82af5bcf5f653597" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "068026e9db2c411a82af5bcf5f653597" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33955 } } }
I20260928 16:51:08.240628 20055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 068026e9db2c411a82af5bcf5f653597 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "068026e9db2c411a82af5bcf5f653597" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "068026e9db2c411a82af5bcf5f653597" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 33955 } } }
I20260928 16:51:08.241117 20056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 068026e9db2c411a82af5bcf5f653597 [sys.catalog]: This master's current role is: LEADER
I20260928 16:51:08.241117 20055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 068026e9db2c411a82af5bcf5f653597 [sys.catalog]: This master's current role is: LEADER
I20260928 16:51:08.242013 20060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:51:08.248281 20060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:51:08.259013 20060 catalog_manager.cc:1347] Generated new cluster ID: 769fb74d369b4b57922833f68c25fea4
I20260928 16:51:08.259150 20060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:51:08.296550 20060 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:51:08.298273 20060 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:51:08.307523 20060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 068026e9db2c411a82af5bcf5f653597: Generated new TSK 0
I20260928 16:51:08.308665 20060 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
I20260928 16:51:08.387145 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33955
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:32787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:51:08.767685 20075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:51:08.768029 20075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:51:08.768169 20075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:51:08.776460 20075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:51:08.776609 20075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:51:08.776782 20075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:51:08.787719 20075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:32787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:51:08.789270 20075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:51:08.790851 20075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:51:08.805852 20080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:51:08.805816 20081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:51:08.805893 20083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:51:08.807009 20075 server_base.cc:1034] running on GCE node
I20260928 16:51:08.807924 20075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:51:08.809098 20075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:51:08.810340 20075 hybrid_clock.cc:648] HybridClock initialized: now 1790614268810245 us; error 85 us; skew 500 ppm
I20260928 16:51:08.810726 20075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:51:08 dist-test-slave-wzcb krb5kdc[19969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614268, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:51:08.814483 20075 init.cc:377] Logged in from keytab as kudu/127.9.93.1@KRBTEST.COM (short username kudu)
I20260928 16:51:08.816370 20075 webserver.cc:460] Webserver started at http://127.9.93.1:39887/ using document root <none> and password file <none>
I20260928 16:51:08.817214 20075 fs_manager.cc:359] Metadata directory not provided
I20260928 16:51:08.817370 20075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:51:08.817701 20075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:51:08.821298 20075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "33845b7a93ce4d8a95a086da91bc76ba"
format_stamp: "Formatted at 2026-09-28 16:51:08 on dist-test-slave-wzcb"
server_key: "11bffcb890cc44a8ce6a8291827dbb0c"
server_key_iv: "e58c5b255e2955864d804e6b3a248c77"
server_key_version: "encryptionkey@0"
I20260928 16:51:08.822234 20075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "33845b7a93ce4d8a95a086da91bc76ba"
format_stamp: "Formatted at 2026-09-28 16:51:08 on dist-test-slave-wzcb"
server_key: "11bffcb890cc44a8ce6a8291827dbb0c"
server_key_iv: "e58c5b255e2955864d804e6b3a248c77"
server_key_version: "encryptionkey@0"
I20260928 16:51:08.828765 20075 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 16:51:08.833246 20090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:08.835171 20075 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:51:08.835422 20075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "33845b7a93ce4d8a95a086da91bc76ba"
format_stamp: "Formatted at 2026-09-28 16:51:08 on dist-test-slave-wzcb"
server_key: "11bffcb890cc44a8ce6a8291827dbb0c"
server_key_iv: "e58c5b255e2955864d804e6b3a248c77"
server_key_version: "encryptionkey@0"
I20260928 16:51:08.835665 20075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:51:08.856310 20075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:51:08.861157 20075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:51:08.861553 20075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:51:08.862687 20075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:51:08.864408 20075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:51:08.864494 20075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:08.864629 20075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:51:08.864687 20075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:08.901340 20075 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:42155
I20260928 16:51:08.901389 20203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:42155 every 8 connection(s)
I20260928 16:51:08.902873 20075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:51:08.905364 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 20075
I20260928 16:51:08.905596 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:51:08.906266 9588 external_mini_cluster.cc:1428] Setting key 3b95d692bae66e82e440a8bba8579126
Sep 28 16:51:08 dist-test-slave-wzcb krb5kdc[19969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614268, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:51:08.928804 20207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:08.907039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:36953 (local address 127.9.93.62:33955)
0928 16:51:08.907603 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:08.907609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:08.907631 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:51:08.910195 (+ 2564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:08.910200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:08.910275 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:08.910385 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:08.913082 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:08.913901 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:08.916428 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:08.916900 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:08.917036 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:08.922030 (+ 4994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:51:08.922037 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:51:08.922040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:51:08.922085 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:51:08.924772 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:51:08.925872 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:51:08.925877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:51:08.925880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:51:08.926044 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:51:08.926649 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:51:08.926654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:51:08.926656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:51:08.926967 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:51:08.927091 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:08.928243 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0928 16:51:08.928514 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":991,"threads_started":1}
I20260928 16:51:08.930320 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:08.906436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33955 (local address 127.9.93.1:36953)
0928 16:51:08.907433 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0928 16:51:08.907493 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 16:51:08.909776 (+ 2283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:51:08.910748 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:51:08.910773 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:51:08.911742 (+ 969us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:51:08.912819 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:51:08.912855 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:51:08.914101 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:51:08.914106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:51:08.916177 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:51:08.916200 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:51:08.917214 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:51:08.917223 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:51:08.917633 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:08.918543 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:51:08.918585 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:51:08.921710 (+ 3125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:51:08.924975 (+ 3265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:51:08.924987 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:51:08.925008 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:51:08.925693 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:51:08.926190 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:51:08.926195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:51:08.926198 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:51:08.926509 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:51:08.927108 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:51:08.927129 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:51:08.927743 (+ 614us) client_negotiation.cc:769] Sending connection context
0928 16:51:08.928083 (+ 340us) client_negotiation.cc:240] Negotiation successful
0928 16:51:08.928770 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":663,"threads_started":1}
I20260928 16:51:08.932349 20204 heartbeater.cc:347] Connected to a master server at 127.9.93.62:33955
I20260928 16:51:08.932770 20204 heartbeater.cc:464] Registering TS with master...
I20260928 16:51:08.933743 20204 heartbeater.cc:511] Master 127.9.93.62:33955 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.93.2@KRBTEST.COM; defaulting to no policy
I20260928 16:51:08.936764 20014 ts_manager.cc:194] Registered new tserver with Master: 33845b7a93ce4d8a95a086da91bc76ba (127.9.93.1:42155)
I20260928 16:51:08.938530 20014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.1@KRBTEST.COM'} at 127.9.93.1:36953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
I20260928 16:51:08.971220 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33955
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:32787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:51:09.352167 20210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:51:09.352555 20210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:51:09.352694 20210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:51:09.362221 20210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:51:09.362389 20210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:51:09.362644 20210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:51:09.373980 20210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:32787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:51:09.375612 20210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:51:09.377225 20210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:51:09.392024 20215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:51:09.392114 20216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:51:09.392050 20218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:51:09.392905 20210 server_base.cc:1034] running on GCE node
I20260928 16:51:09.393661 20210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:51:09.394860 20210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:51:09.396102 20210 hybrid_clock.cc:648] HybridClock initialized: now 1790614269396015 us; error 74 us; skew 500 ppm
I20260928 16:51:09.396513 20210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:51:09 dist-test-slave-wzcb krb5kdc[19969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614269, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:51:09.400240 20210 init.cc:377] Logged in from keytab as kudu/127.9.93.2@KRBTEST.COM (short username kudu)
I20260928 16:51:09.402240 20210 webserver.cc:460] Webserver started at http://127.9.93.2:46751/ using document root <none> and password file <none>
I20260928 16:51:09.403229 20210 fs_manager.cc:359] Metadata directory not provided
I20260928 16:51:09.403429 20210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:51:09.403803 20210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:51:09.407604 20210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "88274446f37949cea6d2b5477b661848"
format_stamp: "Formatted at 2026-09-28 16:51:09 on dist-test-slave-wzcb"
server_key: "1174aac91c15dad6e4a39db229934129"
server_key_iv: "31c59d6f3b28e1813234c99c684c2c1f"
server_key_version: "encryptionkey@0"
I20260928 16:51:09.408711 20210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "88274446f37949cea6d2b5477b661848"
format_stamp: "Formatted at 2026-09-28 16:51:09 on dist-test-slave-wzcb"
server_key: "1174aac91c15dad6e4a39db229934129"
server_key_iv: "31c59d6f3b28e1813234c99c684c2c1f"
server_key_version: "encryptionkey@0"
I20260928 16:51:09.415268 20210 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 16:51:09.419941 20225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:09.421829 20210 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260928 16:51:09.422065 20210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "88274446f37949cea6d2b5477b661848"
format_stamp: "Formatted at 2026-09-28 16:51:09 on dist-test-slave-wzcb"
server_key: "1174aac91c15dad6e4a39db229934129"
server_key_iv: "31c59d6f3b28e1813234c99c684c2c1f"
server_key_version: "encryptionkey@0"
I20260928 16:51:09.422317 20210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:51:09.437310 20210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:51:09.443363 20210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:51:09.443713 20210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:51:09.444878 20210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:51:09.446663 20210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:51:09.446758 20210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:09.446898 20210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:51:09.446961 20210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:09.481829 20210 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:33711
I20260928 16:51:09.481931 20337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:33711 every 8 connection(s)
I20260928 16:51:09.483280 20210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:51:09.489836 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 20210
I20260928 16:51:09.490073 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:51:09.490538 9588 external_mini_cluster.cc:1428] Setting key 3b5e80e3363ff0fcce89b79803b96b03
Sep 28 16:51:09 dist-test-slave-wzcb krb5kdc[19969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614269, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:51:09.507261 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:09.486217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:37811 (local address 127.9.93.62:33955)
0928 16:51:09.486717 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:09.486721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:09.487956 (+ 1235us) server_negotiation.cc:407] Connection header received
0928 16:51:09.490103 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:09.490110 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:09.490214 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:09.490320 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:09.492514 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:09.493613 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:09.495518 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:09.495977 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:09.496104 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:09.500178 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:51:09.500184 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:51:09.500188 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:51:09.500229 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:51:09.503358 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:51:09.504373 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:51:09.504379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:51:09.504383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:51:09.504578 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:51:09.505111 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:51:09.505116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:51:09.505119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:51:09.505540 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:51:09.505688 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:09.506603 (+ 915us) server_negotiation.cc:299] Negotiation successful
0928 16:51:09.506870 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":252,"threads_started":1}
I20260928 16:51:09.508517 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:09.486880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33955 (local address 127.9.93.2:37811)
0928 16:51:09.487759 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0928 16:51:09.487812 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 16:51:09.489696 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:51:09.490647 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:51:09.490661 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:51:09.491292 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:51:09.492216 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:51:09.492247 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:51:09.493830 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:51:09.493834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:51:09.495263 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:51:09.495279 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:51:09.496267 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:51:09.496276 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:51:09.496547 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:09.497302 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:51:09.497330 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:51:09.499863 (+ 2533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:51:09.503605 (+ 3742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:51:09.503614 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:51:09.503632 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:51:09.504197 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:51:09.504716 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:51:09.504720 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:51:09.504722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:51:09.504976 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:51:09.505691 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:51:09.505698 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:51:09.506143 (+ 445us) client_negotiation.cc:769] Sending connection context
0928 16:51:09.506449 (+ 306us) client_negotiation.cc:240] Negotiation successful
0928 16:51:09.506977 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":364,"threads_started":1}
I20260928 16:51:09.510392 20338 heartbeater.cc:347] Connected to a master server at 127.9.93.62:33955
I20260928 16:51:09.510802 20338 heartbeater.cc:464] Registering TS with master...
I20260928 16:51:09.511812 20338 heartbeater.cc:511] Master 127.9.93.62:33955 requested a full tablet report, sending...
I20260928 16:51:09.513720 20014 ts_manager.cc:194] Registered new tserver with Master: 88274446f37949cea6d2b5477b661848 (127.9.93.2:33711)
I20260928 16:51:09.514695 20014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.2@KRBTEST.COM'} at 127.9.93.2:37811
WARNING: no policy specified for kudu/127.9.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
I20260928 16:51:09.551184 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33955
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:32787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:51:09.935056 20344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:51:09.935480 20344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:51:09.935616 20344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 16:51:09.942862 20204 heartbeater.cc:503] Master 127.9.93.62:33955 was elected leader, sending a full tablet report...
W20260928 16:51:09.944103 20344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:51:09.944271 20344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:51:09.944490 20344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:51:09.955873 20344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:32787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:33955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:51:09.957444 20344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:51:09.958972 20344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:51:09.974123 20349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:51:09.974063 20350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:51:09.974098 20352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:51:09.975509 20344 server_base.cc:1034] running on GCE node
I20260928 16:51:09.976266 20344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:51:09.977424 20344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:51:09.978655 20344 hybrid_clock.cc:648] HybridClock initialized: now 1790614269978574 us; error 66 us; skew 500 ppm
I20260928 16:51:09.979020 20344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:51:09 dist-test-slave-wzcb krb5kdc[19969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614269, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:51:09.982556 20344 init.cc:377] Logged in from keytab as kudu/127.9.93.3@KRBTEST.COM (short username kudu)
I20260928 16:51:09.984268 20344 webserver.cc:460] Webserver started at http://127.9.93.3:34575/ using document root <none> and password file <none>
I20260928 16:51:09.985071 20344 fs_manager.cc:359] Metadata directory not provided
I20260928 16:51:09.985217 20344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:51:09.985533 20344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:51:09.989254 20344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "8dbbd80e15e0467fab84142b036b9572"
format_stamp: "Formatted at 2026-09-28 16:51:09 on dist-test-slave-wzcb"
server_key: "313150436dca4b39a0a2cb1ae3df1b1e"
server_key_iv: "7ba3832b564ab43768a58c26bd6d0744"
server_key_version: "encryptionkey@0"
I20260928 16:51:09.990332 20344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "8dbbd80e15e0467fab84142b036b9572"
format_stamp: "Formatted at 2026-09-28 16:51:09 on dist-test-slave-wzcb"
server_key: "313150436dca4b39a0a2cb1ae3df1b1e"
server_key_iv: "7ba3832b564ab43768a58c26bd6d0744"
server_key_version: "encryptionkey@0"
I20260928 16:51:09.997098 20344 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 16:51:10.001811 20359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:10.003749 20344 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:51:10.003947 20344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "8dbbd80e15e0467fab84142b036b9572"
format_stamp: "Formatted at 2026-09-28 16:51:09 on dist-test-slave-wzcb"
server_key: "313150436dca4b39a0a2cb1ae3df1b1e"
server_key_iv: "7ba3832b564ab43768a58c26bd6d0744"
server_key_version: "encryptionkey@0"
I20260928 16:51:10.004191 20344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:51:10.020730 20344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:51:10.025352 20344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:51:10.025636 20344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:51:10.026772 20344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:51:10.028484 20344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:51:10.028575 20344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:10.028677 20344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:51:10.028726 20344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:51:10.063951 20344 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:37515
I20260928 16:51:10.063998 20471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:37515 every 8 connection(s)
I20260928 16:51:10.065392 20344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:51:10.070688 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 20344
I20260928 16:51:10.070889 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:51:10.071367 9588 external_mini_cluster.cc:1428] Setting key 1b1b7a6947e061138a88e130c9f53134
Sep 28 16:51:10 dist-test-slave-wzcb krb5kdc[19969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614269, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:51:10.089490 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:10.068330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:43789 (local address 127.9.93.62:33955)
0928 16:51:10.068854 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:10.068859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:10.069828 (+ 969us) server_negotiation.cc:407] Connection header received
0928 16:51:10.072168 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:10.072176 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:10.072304 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:10.072439 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:10.074887 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.075913 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.077750 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.078223 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.078337 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:10.082489 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:51:10.082496 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:51:10.082500 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:51:10.082552 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:51:10.085452 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:51:10.086585 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:51:10.086592 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:51:10.086596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:51:10.086786 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:51:10.087445 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:51:10.087451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:51:10.087455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:51:10.087846 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:51:10.088028 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:10.088923 (+ 895us) server_negotiation.cc:299] Negotiation successful
0928 16:51:10.089150 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":260,"threads_started":1}
I20260928 16:51:10.090626 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:10.068801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:33955 (local address 127.9.93.3:43789)
0928 16:51:10.069627 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0928 16:51:10.069678 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 16:51:10.071746 (+ 2068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:51:10.072842 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:51:10.072857 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:51:10.073669 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:51:10.074659 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:51:10.074692 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.076095 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.076100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:51:10.077556 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:51:10.077575 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.078495 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.078501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:51:10.078763 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:10.079594 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:51:10.079622 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:51:10.082178 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:51:10.085679 (+ 3501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:51:10.085689 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:51:10.085710 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:51:10.086414 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:51:10.086926 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:51:10.086930 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:51:10.086933 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:51:10.087225 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:51:10.088006 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:51:10.088031 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:51:10.088508 (+ 477us) client_negotiation.cc:769] Sending connection context
0928 16:51:10.088775 (+ 267us) client_negotiation.cc:240] Negotiation successful
0928 16:51:10.089304 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":314,"threads_started":1}
I20260928 16:51:10.092459 20472 heartbeater.cc:347] Connected to a master server at 127.9.93.62:33955
I20260928 16:51:10.092839 20472 heartbeater.cc:464] Registering TS with master...
I20260928 16:51:10.093905 20472 heartbeater.cc:511] Master 127.9.93.62:33955 requested a full tablet report, sending...
I20260928 16:51:10.095932 20014 ts_manager.cc:194] Registered new tserver with Master: 8dbbd80e15e0467fab84142b036b9572 (127.9.93.3:37515)
I20260928 16:51:10.096896 20014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.3@KRBTEST.COM'} at 127.9.93.3:43789
I20260928 16:51:10.101615 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:51:10.121021 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:10.106682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39566 (local address 127.9.93.62:33955)
0928 16:51:10.106886 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:10.106894 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:10.107455 (+ 561us) server_negotiation.cc:407] Connection header received
0928 16:51:10.107693 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:10.107699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:10.107809 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:10.107904 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:10.109557 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.110328 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.111783 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.112231 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.112359 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:10.114923 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:51:10.114929 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:51:10.114932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:51:10.114979 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:51:10.117813 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:51:10.118642 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:51:10.118647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:51:10.118649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:51:10.118800 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:51:10.119243 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:51:10.119247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:51:10.119249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:51:10.119692 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:51:10.119943 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:10.120396 (+ 453us) server_negotiation.cc:299] Negotiation successful
0928 16:51:10.120686 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:51:10.126262 20014 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:51:10.134179 20014 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:51:10.160650 20488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:10.149269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57310 (local address 127.9.93.1:42155)
0928 16:51:10.151263 (+ 1994us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:10.151274 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:10.151328 (+ 54us) server_negotiation.cc:407] Connection header received
0928 16:51:10.151451 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:10.151459 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:10.151736 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:10.151955 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:51:10.152656 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.154164 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.157059 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.158662 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.158807 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:10.159083 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:10.160118 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0928 16:51:10.160324 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1796,"thread_start_us":1999,"threads_started":1}
I20260928 16:51:10.160686 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:10.148548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:37515 (local address 127.0.0.1:47196)
0928 16:51:10.149299 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0928 16:51:10.149355 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 16:51:10.149581 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:51:10.150991 (+ 1410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:51:10.150998 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:51:10.151041 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:51:10.151535 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:51:10.151552 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.154042 (+ 2490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.154048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:51:10.156838 (+ 2790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:51:10.156859 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.159330 (+ 2471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.159337 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:51:10.159671 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:10.159773 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 16:51:10.159924 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 16:51:10.160204 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":77,"thread_start_us":322,"threads_started":1}
I20260928 16:51:10.161697 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:10.148973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47196 (local address 127.9.93.3:37515)
0928 16:51:10.150192 (+ 1219us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:10.150203 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:10.150246 (+ 43us) server_negotiation.cc:407] Connection header received
0928 16:51:10.150374 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:10.150383 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:10.150728 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:10.151001 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:51:10.151732 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.153838 (+ 2106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.157083 (+ 3245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.158663 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.158806 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:10.159085 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:10.161141 (+ 2056us) server_negotiation.cc:299] Negotiation successful
0928 16:51:10.161361 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1026,"thread_start_us":831,"threads_started":1}
I20260928 16:51:10.162379 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:10.148598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:42155 (local address 127.0.0.1:57310)
0928 16:51:10.150492 (+ 1894us) negotiation.cc:107] Waiting for socket to connect
0928 16:51:10.150511 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:51:10.150637 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:51:10.151952 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:51:10.151957 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:51:10.151985 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:51:10.152406 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:51:10.152421 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.154346 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.154352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:51:10.156839 (+ 2487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:51:10.156859 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.159286 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.159296 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:51:10.159675 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:10.159773 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 16:51:10.159924 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 16:51:10.162079 (+ 2155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1764,"mutex_wait_us":2002,"thread_start_us":1579,"threads_started":1}
I20260928 16:51:10.166935 20407 tablet_service.cc:1467] Processing CreateTablet for tablet 57f93f761f0c4316bbe40d982e91ce20 (DEFAULT_TABLE table=test-table [id=f3e15ed327394eb28cf641a3588b82f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:51:10.167654 20138 tablet_service.cc:1467] Processing CreateTablet for tablet 57f93f761f0c4316bbe40d982e91ce20 (DEFAULT_TABLE table=test-table [id=f3e15ed327394eb28cf641a3588b82f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:51:10.169265 20407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57f93f761f0c4316bbe40d982e91ce20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:51:10.169539 20138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57f93f761f0c4316bbe40d982e91ce20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:51:10.171459 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:10.152252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:33711 (local address 127.0.0.1:35532)
0928 16:51:10.162582 (+ 10330us) negotiation.cc:107] Waiting for socket to connect
0928 16:51:10.162603 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:51:10.162733 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:51:10.163771 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:51:10.163776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:51:10.163806 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:51:10.164009 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:51:10.164022 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.166021 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.166027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:51:10.167928 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:51:10.167950 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.170522 (+ 2572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.170529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:51:10.170740 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:10.170778 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 16:51:10.170883 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 16:51:10.171069 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8610,"thread_start_us":11335,"threads_started":1}
I20260928 16:51:10.172333 20490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:10.152688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35532 (local address 127.9.93.2:33711)
0928 16:51:10.162618 (+ 9930us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:10.162628 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:10.162663 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:51:10.162925 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:10.162933 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:10.163199 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:10.163456 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:51:10.164195 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.165818 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.168176 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.170003 (+ 1827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.170161 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:10.170392 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:10.171768 (+ 1376us) server_negotiation.cc:299] Negotiation successful
0928 16:51:10.171997 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9719,"thread_start_us":10446,"threads_started":1}
I20260928 16:51:10.180430 20273 tablet_service.cc:1467] Processing CreateTablet for tablet 57f93f761f0c4316bbe40d982e91ce20 (DEFAULT_TABLE table=test-table [id=f3e15ed327394eb28cf641a3588b82f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:51:10.182353 20273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57f93f761f0c4316bbe40d982e91ce20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:51:10.184716 20491 tablet_bootstrap.cc:492] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba: Bootstrap starting.
I20260928 16:51:10.189356 20491 tablet_bootstrap.cc:654] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba: Neither blocks nor log segments found. Creating new log.
I20260928 16:51:10.191107 20491 log.cc:824] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba: Log is configured to *not* fsync() on all Append() calls
I20260928 16:51:10.193853 20492 tablet_bootstrap.cc:492] T 57f93f761f0c4316bbe40d982e91ce20 P 8dbbd80e15e0467fab84142b036b9572: Bootstrap starting.
I20260928 16:51:10.203055 20492 tablet_bootstrap.cc:654] T 57f93f761f0c4316bbe40d982e91ce20 P 8dbbd80e15e0467fab84142b036b9572: Neither blocks nor log segments found. Creating new log.
I20260928 16:51:10.207131 20492 log.cc:824] T 57f93f761f0c4316bbe40d982e91ce20 P 8dbbd80e15e0467fab84142b036b9572: Log is configured to *not* fsync() on all Append() calls
I20260928 16:51:10.210989 20494 tablet_bootstrap.cc:492] T 57f93f761f0c4316bbe40d982e91ce20 P 88274446f37949cea6d2b5477b661848: Bootstrap starting.
I20260928 16:51:10.211870 20491 tablet_bootstrap.cc:492] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba: No bootstrap required, opened a new log
I20260928 16:51:10.212239 20491 ts_tablet_manager.cc:1397] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.011s
I20260928 16:51:10.213004 20492 tablet_bootstrap.cc:492] T 57f93f761f0c4316bbe40d982e91ce20 P 8dbbd80e15e0467fab84142b036b9572: No bootstrap required, opened a new log
I20260928 16:51:10.213297 20492 ts_tablet_manager.cc:1397] T 57f93f761f0c4316bbe40d982e91ce20 P 8dbbd80e15e0467fab84142b036b9572: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.000s
I20260928 16:51:10.215515 20494 tablet_bootstrap.cc:654] T 57f93f761f0c4316bbe40d982e91ce20 P 88274446f37949cea6d2b5477b661848: Neither blocks nor log segments found. Creating new log.
I20260928 16:51:10.217231 20494 log.cc:824] T 57f93f761f0c4316bbe40d982e91ce20 P 88274446f37949cea6d2b5477b661848: Log is configured to *not* fsync() on all Append() calls
I20260928 16:51:10.221315 20494 tablet_bootstrap.cc:492] T 57f93f761f0c4316bbe40d982e91ce20 P 88274446f37949cea6d2b5477b661848: No bootstrap required, opened a new log
I20260928 16:51:10.221699 20494 ts_tablet_manager.cc:1397] T 57f93f761f0c4316bbe40d982e91ce20 P 88274446f37949cea6d2b5477b661848: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260928 16:51:10.222004 20492 raft_consensus.cc:348] T 57f93f761f0c4316bbe40d982e91ce20 P 8dbbd80e15e0467fab84142b036b9572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbbd80e15e0467fab84142b036b9572" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37515 } } peers { permanent_uuid: "33845b7a93ce4d8a95a086da91bc76ba" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 42155 } } peers { permanent_uuid: "88274446f37949cea6d2b5477b661848" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33711 } }
I20260928 16:51:10.222437 20492 raft_consensus.cc:374] T 57f93f761f0c4316bbe40d982e91ce20 P 8dbbd80e15e0467fab84142b036b9572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:51:10.222595 20492 raft_consensus.cc:729] T 57f93f761f0c4316bbe40d982e91ce20 P 8dbbd80e15e0467fab84142b036b9572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dbbd80e15e0467fab84142b036b9572, State: Initialized, Role: FOLLOWER
I20260928 16:51:10.223258 20492 consensus_queue.cc:260] T 57f93f761f0c4316bbe40d982e91ce20 P 8dbbd80e15e0467fab84142b036b9572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbbd80e15e0467fab84142b036b9572" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37515 } } peers { permanent_uuid: "33845b7a93ce4d8a95a086da91bc76ba" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 42155 } } peers { permanent_uuid: "88274446f37949cea6d2b5477b661848" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33711 } }
I20260928 16:51:10.223125 20491 raft_consensus.cc:348] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbbd80e15e0467fab84142b036b9572" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37515 } } peers { permanent_uuid: "33845b7a93ce4d8a95a086da91bc76ba" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 42155 } } peers { permanent_uuid: "88274446f37949cea6d2b5477b661848" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33711 } }
I20260928 16:51:10.223701 20491 raft_consensus.cc:374] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:51:10.223857 20491 raft_consensus.cc:729] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33845b7a93ce4d8a95a086da91bc76ba, State: Initialized, Role: FOLLOWER
I20260928 16:51:10.224531 20472 heartbeater.cc:503] Master 127.9.93.62:33955 was elected leader, sending a full tablet report...
I20260928 16:51:10.224774 20491 consensus_queue.cc:260] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbbd80e15e0467fab84142b036b9572" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37515 } } peers { permanent_uuid: "33845b7a93ce4d8a95a086da91bc76ba" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 42155 } } peers { permanent_uuid: "88274446f37949cea6d2b5477b661848" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33711 } }
I20260928 16:51:10.225303 20492 ts_tablet_manager.cc:1428] T 57f93f761f0c4316bbe40d982e91ce20 P 8dbbd80e15e0467fab84142b036b9572: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260928 16:51:10.228924 20491 ts_tablet_manager.cc:1428] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260928 16:51:10.233036 20494 raft_consensus.cc:348] T 57f93f761f0c4316bbe40d982e91ce20 P 88274446f37949cea6d2b5477b661848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbbd80e15e0467fab84142b036b9572" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37515 } } peers { permanent_uuid: "33845b7a93ce4d8a95a086da91bc76ba" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 42155 } } peers { permanent_uuid: "88274446f37949cea6d2b5477b661848" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33711 } }
I20260928 16:51:10.233630 20494 raft_consensus.cc:374] T 57f93f761f0c4316bbe40d982e91ce20 P 88274446f37949cea6d2b5477b661848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:51:10.233749 20494 raft_consensus.cc:729] T 57f93f761f0c4316bbe40d982e91ce20 P 88274446f37949cea6d2b5477b661848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88274446f37949cea6d2b5477b661848, State: Initialized, Role: FOLLOWER
I20260928 16:51:10.234416 20494 consensus_queue.cc:260] T 57f93f761f0c4316bbe40d982e91ce20 P 88274446f37949cea6d2b5477b661848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbbd80e15e0467fab84142b036b9572" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37515 } } peers { permanent_uuid: "33845b7a93ce4d8a95a086da91bc76ba" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 42155 } } peers { permanent_uuid: "88274446f37949cea6d2b5477b661848" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33711 } }
I20260928 16:51:10.235700 20338 heartbeater.cc:503] Master 127.9.93.62:33955 was elected leader, sending a full tablet report...
I20260928 16:51:10.236341 20494 ts_tablet_manager.cc:1428] T 57f93f761f0c4316bbe40d982e91ce20 P 88274446f37949cea6d2b5477b661848: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260928 16:51:10.241869 20498 raft_consensus.cc:482] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:51:10.242200 20498 raft_consensus.cc:504] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbbd80e15e0467fab84142b036b9572" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37515 } } peers { permanent_uuid: "33845b7a93ce4d8a95a086da91bc76ba" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 42155 } } peers { permanent_uuid: "88274446f37949cea6d2b5477b661848" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33711 } }
I20260928 16:51:10.244195 20498 leader_election.cc:288] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dbbd80e15e0467fab84142b036b9572 (127.9.93.3:37515), 88274446f37949cea6d2b5477b661848 (127.9.93.2:33711)
I20260928 16:51:10.254089 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:10.244589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:45293 (local address 127.9.93.3:37515)
0928 16:51:10.244831 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:10.244838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:10.245628 (+ 790us) server_negotiation.cc:407] Connection header received
0928 16:51:10.245944 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:10.245952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:10.246051 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:10.246128 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:51:10.247079 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.247965 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.251457 (+ 3492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.252784 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.252945 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:10.253054 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:10.253627 (+ 573us) server_negotiation.cc:299] Negotiation successful
0928 16:51:10.253782 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 16:51:10.254088 20490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:10.244776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:50753 (local address 127.9.93.2:33711)
0928 16:51:10.244984 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:10.244988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:10.245738 (+ 750us) server_negotiation.cc:407] Connection header received
0928 16:51:10.245924 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:10.245935 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:10.246023 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:10.246124 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:51:10.247025 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.247977 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.251495 (+ 3518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.252772 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.252935 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:10.253052 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:10.253627 (+ 575us) server_negotiation.cc:299] Negotiation successful
0928 16:51:10.253782 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:51:10.254384 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:10.244406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:37515 (local address 127.9.93.1:45293)
0928 16:51:10.245474 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0928 16:51:10.245509 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 16:51:10.245694 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:51:10.246374 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:51:10.246379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:51:10.246422 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:51:10.246903 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:51:10.246917 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.248155 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.248161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:51:10.251154 (+ 2993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:51:10.251179 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.253094 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.253103 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:51:10.253395 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:10.253460 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 16:51:10.253568 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 16:51:10.253780 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":272,"thread_start_us":640,"threads_started":1}
I20260928 16:51:10.254534 20501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:10.244803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:33711 (local address 127.9.93.1:50753)
0928 16:51:10.245607 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0928 16:51:10.245655 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 16:51:10.245788 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:51:10.246185 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:51:10.246190 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:51:10.246219 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:51:10.246772 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:51:10.246786 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.248171 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.248176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:51:10.251169 (+ 2993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:51:10.251180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.253122 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.253128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:51:10.253453 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:10.253480 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 16:51:10.253567 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 16:51:10.253780 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":213,"thread_start_us":483,"threads_started":1}
I20260928 16:51:10.255541 20427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f93f761f0c4316bbe40d982e91ce20" candidate_uuid: "33845b7a93ce4d8a95a086da91bc76ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dbbd80e15e0467fab84142b036b9572" is_pre_election: true
I20260928 16:51:10.255571 20293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f93f761f0c4316bbe40d982e91ce20" candidate_uuid: "33845b7a93ce4d8a95a086da91bc76ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88274446f37949cea6d2b5477b661848" is_pre_election: true
I20260928 16:51:10.256107 20293 raft_consensus.cc:2454] T 57f93f761f0c4316bbe40d982e91ce20 P 88274446f37949cea6d2b5477b661848 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33845b7a93ce4d8a95a086da91bc76ba in term 0.
I20260928 16:51:10.256105 20427 raft_consensus.cc:2454] T 57f93f761f0c4316bbe40d982e91ce20 P 8dbbd80e15e0467fab84142b036b9572 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33845b7a93ce4d8a95a086da91bc76ba in term 0.
I20260928 16:51:10.256814 20092 leader_election.cc:302] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33845b7a93ce4d8a95a086da91bc76ba, 88274446f37949cea6d2b5477b661848; no voters:
I20260928 16:51:10.257246 20498 raft_consensus.cc:2781] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:51:10.257402 20498 raft_consensus.cc:482] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:51:10.257519 20498 raft_consensus.cc:3037] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:51:10.260052 20498 raft_consensus.cc:504] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbbd80e15e0467fab84142b036b9572" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37515 } } peers { permanent_uuid: "33845b7a93ce4d8a95a086da91bc76ba" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 42155 } } peers { permanent_uuid: "88274446f37949cea6d2b5477b661848" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33711 } }
I20260928 16:51:10.260759 20498 leader_election.cc:288] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba [CANDIDATE]: Term 1 election: Requested vote from peers 8dbbd80e15e0467fab84142b036b9572 (127.9.93.3:37515), 88274446f37949cea6d2b5477b661848 (127.9.93.2:33711)
I20260928 16:51:10.261420 20427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f93f761f0c4316bbe40d982e91ce20" candidate_uuid: "33845b7a93ce4d8a95a086da91bc76ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dbbd80e15e0467fab84142b036b9572"
I20260928 16:51:10.261511 20293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f93f761f0c4316bbe40d982e91ce20" candidate_uuid: "33845b7a93ce4d8a95a086da91bc76ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88274446f37949cea6d2b5477b661848"
I20260928 16:51:10.261646 20427 raft_consensus.cc:3037] T 57f93f761f0c4316bbe40d982e91ce20 P 8dbbd80e15e0467fab84142b036b9572 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:51:10.261806 20293 raft_consensus.cc:3037] T 57f93f761f0c4316bbe40d982e91ce20 P 88274446f37949cea6d2b5477b661848 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:51:10.264315 20427 raft_consensus.cc:2454] T 57f93f761f0c4316bbe40d982e91ce20 P 8dbbd80e15e0467fab84142b036b9572 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33845b7a93ce4d8a95a086da91bc76ba in term 1.
I20260928 16:51:10.264595 20293 raft_consensus.cc:2454] T 57f93f761f0c4316bbe40d982e91ce20 P 88274446f37949cea6d2b5477b661848 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33845b7a93ce4d8a95a086da91bc76ba in term 1.
I20260928 16:51:10.264838 20091 leader_election.cc:302] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33845b7a93ce4d8a95a086da91bc76ba, 8dbbd80e15e0467fab84142b036b9572; no voters:
I20260928 16:51:10.265232 20498 raft_consensus.cc:2781] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:51:10.265789 20498 raft_consensus.cc:686] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba [term 1 LEADER]: Becoming Leader. State: Replica: 33845b7a93ce4d8a95a086da91bc76ba, State: Running, Role: LEADER
I20260928 16:51:10.266438 20498 consensus_queue.cc:237] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbbd80e15e0467fab84142b036b9572" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37515 } } peers { permanent_uuid: "33845b7a93ce4d8a95a086da91bc76ba" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 42155 } } peers { permanent_uuid: "88274446f37949cea6d2b5477b661848" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33711 } }
I20260928 16:51:10.271699 20013 catalog_manager.cc:5495] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 33845b7a93ce4d8a95a086da91bc76ba (127.9.93.1). New cstate: current_term: 1 leader_uuid: "33845b7a93ce4d8a95a086da91bc76ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbbd80e15e0467fab84142b036b9572" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33845b7a93ce4d8a95a086da91bc76ba" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 42155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88274446f37949cea6d2b5477b661848" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33711 } health_report { overall_health: UNKNOWN } } }
I20260928 16:51:10.306396 20488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:10.300322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57318 (local address 127.9.93.1:42155)
0928 16:51:10.300575 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:10.300580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:10.300602 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:51:10.300702 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:10.300706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:10.300817 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:10.300973 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:51:10.301357 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.302259 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.303896 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.304385 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.304502 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:10.305098 (+ 596us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:51:10.305771 (+ 673us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:51:10.305905 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:10.306020 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 16:51:10.306112 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260928 16:51:10.316318 20427 raft_consensus.cc:1261] T 57f93f761f0c4316bbe40d982e91ce20 P 8dbbd80e15e0467fab84142b036b9572 [term 1 FOLLOWER]: Refusing update from remote peer 33845b7a93ce4d8a95a086da91bc76ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:51:10.316316 20293 raft_consensus.cc:1261] T 57f93f761f0c4316bbe40d982e91ce20 P 88274446f37949cea6d2b5477b661848 [term 1 FOLLOWER]: Refusing update from remote peer 33845b7a93ce4d8a95a086da91bc76ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260928 16:51:10.317103 20473 tablet.cc:2369] T 57f93f761f0c4316bbe40d982e91ce20 P 8dbbd80e15e0467fab84142b036b9572: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:51:10.317489 20502 consensus_queue.cc:1035] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "88274446f37949cea6d2b5477b661848" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 33711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:51:10.317986 20498 consensus_queue.cc:1035] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dbbd80e15e0467fab84142b036b9572" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 37515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:51:10.350744 20014 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:51:10.351171 20014 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:51:10.355538 20014 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 068026e9db2c411a82af5bcf5f653597: Sending DeleteTablet for 3 replicas of tablet 57f93f761f0c4316bbe40d982e91ce20
I20260928 16:51:10.356963 20407 tablet_service.cc:1514] Processing DeleteTablet for tablet 57f93f761f0c4316bbe40d982e91ce20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:51:10 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:47196
I20260928 16:51:10.357087 20273 tablet_service.cc:1514] Processing DeleteTablet for tablet 57f93f761f0c4316bbe40d982e91ce20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:51:10 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:35532
I20260928 16:51:10.357285 20138 tablet_service.cc:1514] Processing DeleteTablet for tablet 57f93f761f0c4316bbe40d982e91ce20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:51:10 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:57310
I20260928 16:51:10.358587 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 20075
I20260928 16:51:10.368265 20517 tablet_replica.cc:330] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba: stopping tablet replica
I20260928 16:51:10.369540 20517 raft_consensus.cc:2229] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:51:10.370620 20517 raft_consensus.cc:2258] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:51:10.371397 20515 tablet_replica.cc:330] T 57f93f761f0c4316bbe40d982e91ce20 P 8dbbd80e15e0467fab84142b036b9572: stopping tablet replica
I20260928 16:51:10.372052 20515 raft_consensus.cc:2229] T 57f93f761f0c4316bbe40d982e91ce20 P 8dbbd80e15e0467fab84142b036b9572 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:51:10.373142 20515 raft_consensus.cc:2258] T 57f93f761f0c4316bbe40d982e91ce20 P 8dbbd80e15e0467fab84142b036b9572 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:51:10.375254 20517 ts_tablet_manager.cc:1905] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:51:10.375680 20515 ts_tablet_manager.cc:1905] T 57f93f761f0c4316bbe40d982e91ce20 P 8dbbd80e15e0467fab84142b036b9572: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:51:10.383934 20515 ts_tablet_manager.cc:1918] T 57f93f761f0c4316bbe40d982e91ce20 P 8dbbd80e15e0467fab84142b036b9572: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:51:10.384303 20515 log.cc:1196] T 57f93f761f0c4316bbe40d982e91ce20 P 8dbbd80e15e0467fab84142b036b9572: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/57f93f761f0c4316bbe40d982e91ce20
I20260928 16:51:10.385025 20515 ts_tablet_manager.cc:1939] T 57f93f761f0c4316bbe40d982e91ce20 P 8dbbd80e15e0467fab84142b036b9572: Deleting consensus metadata
I20260928 16:51:10.387576 19998 catalog_manager.cc:4841] TS 8dbbd80e15e0467fab84142b036b9572 (127.9.93.3:37515): tablet 57f93f761f0c4316bbe40d982e91ce20 (table test-table [id=f3e15ed327394eb28cf641a3588b82f4]) successfully deleted
I20260928 16:51:10.388768 20516 tablet_replica.cc:330] T 57f93f761f0c4316bbe40d982e91ce20 P 88274446f37949cea6d2b5477b661848: stopping tablet replica
I20260928 16:51:10.389482 20516 raft_consensus.cc:2229] T 57f93f761f0c4316bbe40d982e91ce20 P 88274446f37949cea6d2b5477b661848 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:51:10.389971 20517 ts_tablet_manager.cc:1918] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:51:10.390224 20516 raft_consensus.cc:2258] T 57f93f761f0c4316bbe40d982e91ce20 P 88274446f37949cea6d2b5477b661848 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:51:10.390275 20517 log.cc:1196] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-0/wal/wals/57f93f761f0c4316bbe40d982e91ce20
I20260928 16:51:10.390946 20517 ts_tablet_manager.cc:1939] T 57f93f761f0c4316bbe40d982e91ce20 P 33845b7a93ce4d8a95a086da91bc76ba: Deleting consensus metadata
I20260928 16:51:10.393211 19999 catalog_manager.cc:4841] TS 33845b7a93ce4d8a95a086da91bc76ba (127.9.93.1:42155): tablet 57f93f761f0c4316bbe40d982e91ce20 (table test-table [id=f3e15ed327394eb28cf641a3588b82f4]) successfully deleted
I20260928 16:51:10.393528 20516 ts_tablet_manager.cc:1905] T 57f93f761f0c4316bbe40d982e91ce20 P 88274446f37949cea6d2b5477b661848: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 16:51:10 dist-test-slave-wzcb krb5kdc[19969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.1@KRBTEST.COM
I20260928 16:51:10.402386 20516 ts_tablet_manager.cc:1918] T 57f93f761f0c4316bbe40d982e91ce20 P 88274446f37949cea6d2b5477b661848: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:51:10.402685 20516 log.cc:1196] T 57f93f761f0c4316bbe40d982e91ce20 P 88274446f37949cea6d2b5477b661848: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/57f93f761f0c4316bbe40d982e91ce20
I20260928 16:51:10.402697 20488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:10.360773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57328 (local address 127.9.93.1:42155)
0928 16:51:10.361016 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:10.361024 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:10.371423 (+ 10399us) server_negotiation.cc:407] Connection header received
0928 16:51:10.372077 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:10.372084 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:10.372206 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:10.372292 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:10.377442 (+ 5150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.378837 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.389210 (+ 10373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.389945 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.390095 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:10.395906 (+ 5811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:51:10.395923 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:51:10.395939 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:51:10.396002 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:51:10.399270 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:51:10.400155 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:51:10.400164 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:51:10.400167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:51:10.400384 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:51:10.400823 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:51:10.400827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:51:10.400829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:51:10.401259 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:51:10.401579 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:10.401903 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 16:51:10.402193 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 16:51:10.403388 20516 ts_tablet_manager.cc:1939] T 57f93f761f0c4316bbe40d982e91ce20 P 88274446f37949cea6d2b5477b661848: Deleting consensus metadata
I20260928 16:51:10.403506 20199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:51:10.405627 19999 catalog_manager.cc:4841] TS 88274446f37949cea6d2b5477b661848 (127.9.93.2:33711): tablet 57f93f761f0c4316bbe40d982e91ce20 (table test-table [id=f3e15ed327394eb28cf641a3588b82f4]) successfully deleted
I20260928 16:51:10.520807 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 20075
I20260928 16:51:10.538758 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 20210
Sep 28 16:51:10 dist-test-slave-wzcb krb5kdc[19969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.2@KRBTEST.COM
I20260928 16:51:10.556478 20490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:10.539700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.9.93.2:33711)
0928 16:51:10.539894 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:10.539900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:10.539923 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:51:10.540019 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:10.540023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:10.540111 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:10.540220 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:10.541556 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.542473 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.544288 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.544748 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.544871 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:10.549477 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:51:10.549502 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:51:10.549522 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:51:10.549579 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:51:10.552971 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:51:10.553802 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:51:10.553811 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:51:10.553814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:51:10.553995 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:51:10.554501 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:51:10.554507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:51:10.554509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:51:10.555066 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:51:10.555413 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:10.555915 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 16:51:10.556150 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 16:51:10.557411 20333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:51:10.653462 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 20210
I20260928 16:51:10.669323 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 20344
Sep 28 16:51:10 dist-test-slave-wzcb krb5kdc[19969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.3@KRBTEST.COM
I20260928 16:51:10.687718 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:51:10.670301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47212 (local address 127.9.93.3:37515)
0928 16:51:10.670588 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 16:51:10.670595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:51:10.670618 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:51:10.670691 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:51:10.670695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:51:10.670783 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:51:10.670877 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:51:10.672544 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.673369 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.674851 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:51:10.675493 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:51:10.675664 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:51:10.680425 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:51:10.680448 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:51:10.680467 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:51:10.680526 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:51:10.683989 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:51:10.684969 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:51:10.684981 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:51:10.684983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:51:10.685186 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:51:10.685747 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:51:10.685752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:51:10.685754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:51:10.686264 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:51:10.686553 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 16:51:10.687003 (+ 450us) server_negotiation.cc:299] Negotiation successful
0928 16:51:10.687261 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260928 16:51:10.688670 20467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:51:10.784919 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 20344
I20260928 16:51:10.800088 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 19982
I20260928 16:51:10.801422 20044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:51:10.917064 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 19982
2026-09-28T16:51:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4559 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:51:11 dist-test-slave-wzcb krb5kdc[20525](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:51:11 dist-test-slave-wzcb krb5kdc[20525](info): set up 2 sockets
Sep 28 16:51:11 dist-test-slave-wzcb krb5kdc[20525](info): commencing operation
krb5kdc: starting...
W20260928 16:51:11.962566 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.007s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:51:12 dist-test-slave-wzcb krb5kdc[20525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:51:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:51:12Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.93.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.93.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.93.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/rangeradmin_127.9.93.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.93.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/rangeradmin_127.9.93.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.93.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.93.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.93.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/rangerlookup_127.9.93.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.93.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/rangerlookup_127.9.93.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.93.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.93.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.93.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/HTTP_127.9.93.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.93.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/HTTP_127.9.93.20@KRBTEST.COM.keytab.
I20260928 16:51:12.347795 9588 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskkzF10S/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/postgres -l logfile start
2026-09-28 16:51:15.089 UTC [20555] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 16:51:15.089 UTC [20555] LOG: listening on IPv4 address "127.9.93.20", port 43555
2026-09-28 16:51:15.093 UTC [20555] LOG: listening on Unix socket "/tmp/.s.PGSQL.43555"
2026-09-28 16:51:15.116 UTC [20558] LOG: database system was shut down at 2026-09-28 16:51:13 UTC
2026-09-28 16:51:15.120 UTC [20555] LOG: database system is ready to accept connections
I20260928 16:51:16.026213 9588 mini_postgres.cc:96] Postgres bound to 43555
2026-09-28 16:51:16.047 UTC [20566] FATAL: database "slave" does not exist
127.9.93.20:43555 - accepting connections
I20260928 16:51:16.047955 9588 mini_ranger.cc:161] Starting Ranger...
I20260928 16:51:16.081024 9588 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 16:51:16.603739 9588 mini_ranger.cc:87] Created ranger Postgres database
I20260928 16:51:16.603899 9588 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ranger-admin
2026-09-28 16:51:16,927 [I] DB FLAVOR :POSTGRES
2026-09-28 16:51:16,928 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 16:51:16,928 [I] Checking connection..
2026-09-28 16:51:16,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:43555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 16:51:17,204 [I] Checking connection passed.
2026-09-28 16:51:17,204 [I] --------- Verifying version history table ---------
2026-09-28 16:51:17,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:43555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:51:17,454 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 16:51:17,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:43555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:51:17,714 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 16:51:17,715 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 16:51:17,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:43555/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 16:51:17.997 UTC [20630] WARNING: there is no transaction in progress
2026-09-28 16:51:18,005 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 16:51:18,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:43555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:51:18,274 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 16:51:18,274 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 16:51:18,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:43555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 16:51:18,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:43555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 16:51:18,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:43555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wzcb.c.gcp-upstream.internal','N') ;"
2026-09-28 16:51:19,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:43555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 16:51:19,374 [I] Table x_portal_user does not exist in database ranger
2026-09-28 16:51:19,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:43555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 16:51:19,632 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 16:51:19,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:43555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 16:51:19,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:43555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 16:51:20,206 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 16:51:20,206 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:43555/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 16:51:21.382 UTC [20738] WARNING: there is no transaction in progress
2026-09-28 16:51:21.401 UTC [20738] WARNING: there is no transaction in progress
2026-09-28 16:51:21.422 UTC [20738] WARNING: there is no transaction in progress
2026-09-28 16:51:21.441 UTC [20738] WARNING: there is no transaction in progress
2026-09-28 16:51:21.460 UTC [20738] WARNING: there is no transaction in progress
2026-09-28 16:51:21.483 UTC [20738] WARNING: there is no transaction in progress
2026-09-28 16:51:21.552 UTC [20738] WARNING: there is no transaction in progress
2026-09-28 16:51:21.562 UTC [20738] WARNING: there is no transaction in progress
2026-09-28 16:51:21.580 UTC [20738] WARNING: there is no transaction in progress
2026-09-28 16:51:21.592 UTC [20738] WARNING: there is no transaction in progress
2026-09-28 16:51:21.605 UTC [20738] WARNING: there is no transaction in progress
2026-09-28 16:51:21.623 UTC [20738] WARNING: there is no transaction in progress
2026-09-28 16:51:21.636 UTC [20738] WARNING: there is no transaction in progress
2026-09-28 16:51:21.644 UTC [20738] WARNING: there is no transaction in progress
2026-09-28 16:51:21.654 UTC [20738] WARNING: there is no transaction in progress
2026-09-28 16:51:21.663 UTC [20738] WARNING: there is no transaction in progress
2026-09-28 16:51:22,528 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 16:51:22,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:43555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 16:51:22,783 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 16:51:22,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:43555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 16:51:23,077 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 16:51:23,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:43555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 16:51:23,358 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 16:51:23,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:43555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 16:51:23,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:43555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 16:51:23,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:43555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wzcb.c.gcp-upstream.internal';"
2026-09-28 16:51:24,178 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 16:51:24,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:43555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 16:51:24,456 [I] DB_PATCHES have already been applied
I20260928 16:51:24.462908 9588 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ranger-admin:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskkzF10S/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 16:51:24.463017 9588 mini_ranger.cc:193] Using host: 127.9.93.20
I20260928 16:51:24.481534 9588 mini_ranger.cc:227] Ranger admin URL: http://127.9.93.20:44983
Sep 28, 2026 4:51:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:51:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:51:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 4:51:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 4:51:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 4:51:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 4:51:25 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 16:51:25.534103 9588 mini_ranger.cc:160] Time spent starting Ranger: real 9.486s user 0.003s sys 0.055s
Sep 28, 2026 4:51:48 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 4:51:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 4:51:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 4:51:48 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 4:51:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 4:51:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 4:51:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 4:51:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 4:51:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 4:51:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 4:51:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 4:51:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 4:51:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 4:51:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 4:51:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 4:51:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 4:51:48 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 4:51:49 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 16:51:50.549189 9588 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.93.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.93.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.93.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/rangerkms_127.9.93.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.93.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/rangerkms_127.9.93.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.93.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.93.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.93.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/HTTP_127.9.93.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.93.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/HTTP_127.9.93.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 16:51:50 dist-test-slave-wzcb krb5kdc[20525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614310, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 16:51:50.760072 9588 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 16:51:50.786803 9588 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 16:51:51.916595 9588 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 16:51:51.916786 9588 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ranger-kms
I20260928 16:51:51.916936 9588 mini_ranger_kms.cc:227] Using postgres at 127.9.93.20:43555
2026-09-28 16:51:52,963 [I] DB FLAVOR :POSTGRES
2026-09-28 16:51:52,964 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 16:51:52,964 [I] Checking connection
2026-09-28 16:51:52,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:43555/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 16:51:53,253 [I] connection success
2026-09-28 16:51:53,254 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 16:51:53,254 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 16:51:53,254 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:43555/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 16:51:53,526 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 16:51:53,526 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 16:51:53,527 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 16:51:53,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:43555/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 16:51:53,843 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 16:51:54.470857 9588 mini_ranger_kms.cc:310] Created kms service
I20260928 16:51:54.707686 9588 mini_ranger_kms.cc:326] Created kudu user
I20260928 16:51:54.783064 9588 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 16:51:54 dist-test-slave-wzcb krb5kdc[20525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614310, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.93.20@KRBTEST.COM
I20260928 16:51:55.351812 9588 mini_ranger_kms.cc:384] Added ranger policy
I20260928 16:51:55.353873 9588 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ranger-kms:/tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 16:51:55.353978 9588 mini_ranger_kms.cc:240] Using host: 127.9.93.20
I20260928 16:51:55.376072 9588 mini_ranger_kms.cc:276] Ranger KMS PID: 20919
I20260928 16:51:55.376332 9588 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.93.20:47663
Sep 28, 2026 4:51:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:51:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:51:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 4:51:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 4:51:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 4:51:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 16:51:57.571012 9588 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.811s user 0.183s sys 0.486s
I20260928 16:51:57.571476 9588 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 16:51:57.571770 9588 mini_ranger_kms.cc:401] 127.9.93.20:47663/kms/v1/keys
Sep 28 16:51:58 dist-test-slave-wzcb krb5kdc[20525](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790614318, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.93.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 16:51:58 dist-test-slave-wzcb krb5kdc[20525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614318, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.93.20@KRBTEST.COM for HTTP/127.9.93.20@KRBTEST.COM
Sep 28 16:51:59 dist-test-slave-wzcb krb5kdc[20525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614318, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.93.20@KRBTEST.COM for HTTP/127.9.93.20@KRBTEST.COM
Sep 28 16:52:01 dist-test-slave-wzcb krb5kdc[20525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:52:01.941398 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:34143
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:34143
--ranger_config_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.93.20:47663/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:52:02.653606 20969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:52:02.653988 20969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:52:02.654137 20969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:52:02.665171 20969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:52:02.665339 20969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:52:02.665426 20969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:52:02.665508 20969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:52:02.665589 20969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:52:02.681187 20969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.93.20:47663/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:34143
--ranger_config_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:34143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:52:02.683038 20969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:52:02.685014 20969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:52:02.703342 20974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:52:02.703351 20975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:52:02.703467 20977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:52:02.704587 20969 server_base.cc:1034] running on GCE node
I20260928 16:52:02.705639 20969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:52:02.707638 20969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:52:02.708904 20969 hybrid_clock.cc:648] HybridClock initialized: now 1790614322708844 us; error 66 us; skew 500 ppm
I20260928 16:52:02.709339 20969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:52:02 dist-test-slave-wzcb krb5kdc[20525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614322, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:52:02.713801 20969 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:52:02.716446 20969 webserver.cc:460] Webserver started at http://127.9.93.62:36615/ using document root <none> and password file <none>
I20260928 16:52:02.717538 20969 fs_manager.cc:359] Metadata directory not provided
I20260928 16:52:02.717808 20969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:52:02.718254 20969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:52:02 dist-test-slave-wzcb krb5kdc[20525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614322, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for HTTP/127.9.93.20@KRBTEST.COM
I20260928 16:52:02.795897 20969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "b661e3ba844f4311851b718c3f0758b4"
format_stamp: "Formatted at 2026-09-28 16:52:02 on dist-test-slave-wzcb"
server_key: "82db039531b99a147342920311319d80"
server_key_iv: "578f174918bfcbf670df883f6ec01c04"
server_key_version: "kuduclusterkey@0"
I20260928 16:52:02.796896 20969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "b661e3ba844f4311851b718c3f0758b4"
format_stamp: "Formatted at 2026-09-28 16:52:02 on dist-test-slave-wzcb"
server_key: "82db039531b99a147342920311319d80"
server_key_iv: "578f174918bfcbf670df883f6ec01c04"
server_key_version: "kuduclusterkey@0"
I20260928 16:52:02.804446 20969 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20260928 16:52:02.830291 20984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:52:02.832509 20969 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s
I20260928 16:52:02.832841 20969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "b661e3ba844f4311851b718c3f0758b4"
format_stamp: "Formatted at 2026-09-28 16:52:02 on dist-test-slave-wzcb"
server_key: "82db039531b99a147342920311319d80"
server_key_iv: "578f174918bfcbf670df883f6ec01c04"
server_key_version: "kuduclusterkey@0"
I20260928 16:52:02.833266 20969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:52:02.854256 20969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:52:02.860512 20969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:52:02.860993 20969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:52:02.900220 20969 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:34143
I20260928 16:52:02.900262 21035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:34143 every 8 connection(s)
I20260928 16:52:02.902204 20969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:52:02.908221 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 20969
I20260928 16:52:02.908458 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:52:02.909143 21036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:52:02.909277 9588 external_mini_cluster.cc:1428] Setting key a8f129bf1b93b03e5968b8293b1bb7aa
Sep 28 16:52:02 dist-test-slave-wzcb krb5kdc[20525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:52:02.932027 21036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b661e3ba844f4311851b718c3f0758b4: Bootstrap starting.
I20260928 16:52:02.937592 21036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b661e3ba844f4311851b718c3f0758b4: Neither blocks nor log segments found. Creating new log.
I20260928 16:52:02.940266 21036 log.cc:824] T 00000000000000000000000000000000 P b661e3ba844f4311851b718c3f0758b4: Log is configured to *not* fsync() on all Append() calls
I20260928 16:52:02.944689 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:02.912084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45402 (local address 127.9.93.62:34143)
0928 16:52:02.914042 (+ 1958us) server_negotiation.cc:206] Beginning negotiation
0928 16:52:02.914064 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 16:52:02.914166 (+ 102us) server_negotiation.cc:407] Connection header received
0928 16:52:02.917999 (+ 3833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:52:02.918050 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:52:02.918872 (+ 822us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:52:02.919599 (+ 727us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:52:02.921218 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:02.924200 (+ 2982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:02.925771 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:02.927452 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:02.927707 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:02.932475 (+ 4768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:52:02.932493 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:52:02.932520 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:52:02.932634 (+ 114us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:52:02.938421 (+ 5787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:52:02.939408 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:52:02.939418 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:52:02.939427 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:52:02.939697 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:52:02.940221 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:52:02.940227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:52:02.940230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:52:02.941146 (+ 916us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:52:02.941576 (+ 430us) server_negotiation.cc:1035] Waiting for connection context
0928 16:52:02.942190 (+ 614us) server_negotiation.cc:299] Negotiation successful
0928 16:52:02.942622 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1553,"thread_start_us":1094,"threads_started":1}
I20260928 16:52:02.945813 21036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b661e3ba844f4311851b718c3f0758b4: No bootstrap required, opened a new log
I20260928 16:52:02.959275 21036 raft_consensus.cc:348] T 00000000000000000000000000000000 P b661e3ba844f4311851b718c3f0758b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b661e3ba844f4311851b718c3f0758b4" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 34143 } }
I20260928 16:52:02.959828 21036 raft_consensus.cc:374] T 00000000000000000000000000000000 P b661e3ba844f4311851b718c3f0758b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:52:02.959937 21036 raft_consensus.cc:729] T 00000000000000000000000000000000 P b661e3ba844f4311851b718c3f0758b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b661e3ba844f4311851b718c3f0758b4, State: Initialized, Role: FOLLOWER
I20260928 16:52:02.960649 21036 consensus_queue.cc:260] T 00000000000000000000000000000000 P b661e3ba844f4311851b718c3f0758b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b661e3ba844f4311851b718c3f0758b4" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 34143 } }
I20260928 16:52:02.960909 21036 raft_consensus.cc:388] T 00000000000000000000000000000000 P b661e3ba844f4311851b718c3f0758b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:52:02.961051 21036 raft_consensus.cc:482] T 00000000000000000000000000000000 P b661e3ba844f4311851b718c3f0758b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:52:02.961229 21036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b661e3ba844f4311851b718c3f0758b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:52:02.963996 21036 raft_consensus.cc:504] T 00000000000000000000000000000000 P b661e3ba844f4311851b718c3f0758b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b661e3ba844f4311851b718c3f0758b4" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 34143 } }
I20260928 16:52:02.964704 21036 leader_election.cc:302] T 00000000000000000000000000000000 P b661e3ba844f4311851b718c3f0758b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b661e3ba844f4311851b718c3f0758b4; no voters:
I20260928 16:52:02.965631 21036 leader_election.cc:288] T 00000000000000000000000000000000 P b661e3ba844f4311851b718c3f0758b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:52:02.965909 21041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b661e3ba844f4311851b718c3f0758b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:52:02.967834 21041 raft_consensus.cc:686] T 00000000000000000000000000000000 P b661e3ba844f4311851b718c3f0758b4 [term 1 LEADER]: Becoming Leader. State: Replica: b661e3ba844f4311851b718c3f0758b4, State: Running, Role: LEADER
I20260928 16:52:02.968647 21036 sys_catalog.cc:564] T 00000000000000000000000000000000 P b661e3ba844f4311851b718c3f0758b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:52:02.968657 21041 consensus_queue.cc:237] T 00000000000000000000000000000000 P b661e3ba844f4311851b718c3f0758b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b661e3ba844f4311851b718c3f0758b4" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 34143 } }
I20260928 16:52:02.982102 21042 sys_catalog.cc:455] T 00000000000000000000000000000000 P b661e3ba844f4311851b718c3f0758b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b661e3ba844f4311851b718c3f0758b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b661e3ba844f4311851b718c3f0758b4" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 34143 } } }
I20260928 16:52:02.982787 21042 sys_catalog.cc:458] T 00000000000000000000000000000000 P b661e3ba844f4311851b718c3f0758b4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:52:02.982523 21043 sys_catalog.cc:455] T 00000000000000000000000000000000 P b661e3ba844f4311851b718c3f0758b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b661e3ba844f4311851b718c3f0758b4. Latest consensus state: current_term: 1 leader_uuid: "b661e3ba844f4311851b718c3f0758b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b661e3ba844f4311851b718c3f0758b4" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 34143 } } }
I20260928 16:52:02.982935 21043 sys_catalog.cc:458] T 00000000000000000000000000000000 P b661e3ba844f4311851b718c3f0758b4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:52:02.987174 21052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:52:02.995877 21052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:52:02.997007 21036 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 16:52:03.027954 21052 catalog_manager.cc:1347] Generated new cluster ID: b689202eb6514e9294e638075d9b316a
I20260928 16:52:03.028193 21052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:52:03.054549 21052 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:52:03.056519 21052 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:52:03.068668 21052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b661e3ba844f4311851b718c3f0758b4: Generated new TSK 0
I20260928 16:52:03.069940 21052 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
I20260928 16:52:04.064049 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.93.20:47663/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:34143
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:52:04 dist-test-slave-wzcb krb5kdc[20525](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790614324, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 16:52:04.847405 21082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:52:04.847886 21082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:52:04.848086 21082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:52:04.859441 21082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:52:04.859663 21082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:52:04.859889 21082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:52:04.879602 21082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.93.20:47663/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:34143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:52:04.882447 21082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:52:04.884918 21082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:52:04.910769 21092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:52:04.914551 21093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:52:04.937660 21095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:52:04.948092 21082 server_base.cc:1034] running on GCE node
I20260928 16:52:04.949271 21082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:52:04.951242 21082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:52:04.952704 21082 hybrid_clock.cc:648] HybridClock initialized: now 1790614324952510 us; error 115 us; skew 500 ppm
I20260928 16:52:04.953333 21082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:52:04 dist-test-slave-wzcb krb5kdc[20525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614324, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:52:04.958705 21082 init.cc:377] Logged in from keytab as kudu/127.9.93.1@KRBTEST.COM (short username kudu)
I20260928 16:52:04.967914 21082 webserver.cc:460] Webserver started at http://127.9.93.1:37281/ using document root <none> and password file <none>
I20260928 16:52:04.969498 21082 fs_manager.cc:359] Metadata directory not provided
I20260928 16:52:04.969916 21082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:52:04.970441 21082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:52:04 dist-test-slave-wzcb krb5kdc[20525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614324, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for HTTP/127.9.93.20@KRBTEST.COM
I20260928 16:52:05.004446 21082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "cc5e55e26c2b47809f1905424c848bab"
format_stamp: "Formatted at 2026-09-28 16:52:04 on dist-test-slave-wzcb"
server_key: "39af5602373794c6a4e8afebbfcb0f3d"
server_key_iv: "23fd11d5275bd5e176815e59a81ecc5f"
server_key_version: "kuduclusterkey@0"
I20260928 16:52:05.006409 21082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "cc5e55e26c2b47809f1905424c848bab"
format_stamp: "Formatted at 2026-09-28 16:52:04 on dist-test-slave-wzcb"
server_key: "39af5602373794c6a4e8afebbfcb0f3d"
server_key_iv: "23fd11d5275bd5e176815e59a81ecc5f"
server_key_version: "kuduclusterkey@0"
I20260928 16:52:05.039191 21082 fs_manager.cc:691] Time spent creating directory manager: real 0.032s user 0.015s sys 0.015s
I20260928 16:52:05.088913 21102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:52:05.092221 21082 fs_manager.cc:725] Time spent opening block manager: real 0.026s user 0.012s sys 0.010s
I20260928 16:52:05.092626 21082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "cc5e55e26c2b47809f1905424c848bab"
format_stamp: "Formatted at 2026-09-28 16:52:04 on dist-test-slave-wzcb"
server_key: "39af5602373794c6a4e8afebbfcb0f3d"
server_key_iv: "23fd11d5275bd5e176815e59a81ecc5f"
server_key_version: "kuduclusterkey@0"
I20260928 16:52:05.093189 21082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 28 16:52:05 dist-test-slave-wzcb krb5kdc[20525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614324, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for HTTP/127.9.93.20@KRBTEST.COM
I20260928 16:52:05.251902 21082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:52:05.260987 21082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:52:05.261802 21082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:52:05.274995 21082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:52:05.293316 21082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:52:05.293691 21082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.001s
I20260928 16:52:05.294147 21082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:52:05.294399 21082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:52:05 dist-test-slave-wzcb krb5kdc[20525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614324, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for HTTP/127.9.93.20@KRBTEST.COM
I20260928 16:52:06.141531 21082 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:43611
I20260928 16:52:06.141821 21216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:43611 every 8 connection(s)
I20260928 16:52:06.144495 21082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:52:06.148010 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 21082
I20260928 16:52:06.148267 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:52:06.149554 9588 external_mini_cluster.cc:1428] Setting key 13857c281d1dbeec8ec285c195e12517
Sep 28 16:52:06 dist-test-slave-wzcb krb5kdc[20525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614324, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.2@KRBTEST.COM" created.
I20260928 16:52:06.229898 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:06.166491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:34143 (local address 127.9.93.1:55715)
0928 16:52:06.186047 (+ 19556us) negotiation.cc:107] Waiting for socket to connect
0928 16:52:06.186136 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0928 16:52:06.193242 (+ 7106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:52:06.194778 (+ 1536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:52:06.194792 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:52:06.195753 (+ 961us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:52:06.197196 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:52:06.197236 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:52:06.199886 (+ 2650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:52:06.199893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:52:06.204150 (+ 4257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:52:06.204215 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:52:06.207840 (+ 3625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:52:06.207854 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:52:06.208389 (+ 535us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:06.209631 (+ 1242us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:52:06.209670 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:52:06.214893 (+ 5223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:52:06.222381 (+ 7488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:52:06.222394 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:52:06.222418 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:52:06.223482 (+ 1064us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:52:06.224257 (+ 775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:52:06.224263 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:52:06.224266 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:52:06.224861 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:52:06.226011 (+ 1150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:52:06.226021 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:52:06.226613 (+ 592us) client_negotiation.cc:769] Sending connection context
0928 16:52:06.227012 (+ 399us) client_negotiation.cc:240] Negotiation successful
0928 16:52:06.227837 (+ 825us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19014,"thread_start_us":17922,"threads_started":1}
I20260928 16:52:06.231745 21219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:06.165062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:55715 (local address 127.9.93.62:34143)
0928 16:52:06.184662 (+ 19600us) server_negotiation.cc:206] Beginning negotiation
0928 16:52:06.184670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:52:06.189322 (+ 4652us) server_negotiation.cc:407] Connection header received
0928 16:52:06.193752 (+ 4430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:52:06.193760 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:52:06.193888 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:52:06.193996 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:52:06.197525 (+ 3529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:06.199234 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:06.204575 (+ 5341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:06.206239 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:06.206477 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:06.215469 (+ 8992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:52:06.215486 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:52:06.215492 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:52:06.215553 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:52:06.222092 (+ 6539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:52:06.223697 (+ 1605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:52:06.223707 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:52:06.223710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:52:06.224053 (+ 343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:52:06.225141 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:52:06.225147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:52:06.225150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:52:06.225792 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:52:06.230927 (+ 5135us) server_negotiation.cc:1035] Waiting for connection context
0928 16:52:06.231083 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 16:52:06.231435 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19373,"thread_start_us":19527,"threads_started":1}
I20260928 16:52:06.233464 21217 heartbeater.cc:347] Connected to a master server at 127.9.93.62:34143
I20260928 16:52:06.233861 21217 heartbeater.cc:464] Registering TS with master...
I20260928 16:52:06.235329 21217 heartbeater.cc:511] Master 127.9.93.62:34143 requested a full tablet report, sending...
I20260928 16:52:06.238929 21001 ts_manager.cc:194] Registered new tserver with Master: cc5e55e26c2b47809f1905424c848bab (127.9.93.1:43611)
I20260928 16:52:06.241973 21001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.1@KRBTEST.COM'} at 127.9.93.1:55715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
W20260928 16:52:06.258173 9588 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.9.93.2: real 0.104s user 0.000s sys 0.027s
I20260928 16:52:06.259128 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.93.20:47663/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:34143
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:52:07.001300 21230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:52:07.001688 21230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:52:07.001874 21230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:52:07.010919 21230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:52:07.011118 21230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:52:07.011335 21230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:52:07.024921 21230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.93.20:47663/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:34143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:52:07.026857 21230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:52:07.028759 21230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:52:07.047500 21238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:52:07.047564 21236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:52:07.047466 21235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:52:07.048784 21230 server_base.cc:1034] running on GCE node
I20260928 16:52:07.049606 21230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:52:07.050933 21230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:52:07.052239 21230 hybrid_clock.cc:648] HybridClock initialized: now 1790614327052136 us; error 58 us; skew 500 ppm
I20260928 16:52:07.052719 21230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:52:07 dist-test-slave-wzcb krb5kdc[20525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614327, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:52:07.057116 21230 init.cc:377] Logged in from keytab as kudu/127.9.93.2@KRBTEST.COM (short username kudu)
I20260928 16:52:07.059247 21230 webserver.cc:460] Webserver started at http://127.9.93.2:37199/ using document root <none> and password file <none>
I20260928 16:52:07.060292 21230 fs_manager.cc:359] Metadata directory not provided
I20260928 16:52:07.060477 21230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:52:07.060910 21230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:52:07 dist-test-slave-wzcb krb5kdc[20525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614327, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for HTTP/127.9.93.20@KRBTEST.COM
I20260928 16:52:07.081946 21230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "5cb322a7251040d6b7e72f32be6e6f73"
format_stamp: "Formatted at 2026-09-28 16:52:07 on dist-test-slave-wzcb"
server_key: "70bcda88f5198cc53775408ffc3b5eac"
server_key_iv: "6ac087bb4853ca81f7f6bd9dfa4e02e1"
server_key_version: "kuduclusterkey@0"
I20260928 16:52:07.083232 21230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "5cb322a7251040d6b7e72f32be6e6f73"
format_stamp: "Formatted at 2026-09-28 16:52:07 on dist-test-slave-wzcb"
server_key: "70bcda88f5198cc53775408ffc3b5eac"
server_key_iv: "6ac087bb4853ca81f7f6bd9dfa4e02e1"
server_key_version: "kuduclusterkey@0"
I20260928 16:52:07.092176 21230 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260928 16:52:07.113001 21245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:52:07.115335 21230 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
I20260928 16:52:07.115655 21230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "5cb322a7251040d6b7e72f32be6e6f73"
format_stamp: "Formatted at 2026-09-28 16:52:07 on dist-test-slave-wzcb"
server_key: "70bcda88f5198cc53775408ffc3b5eac"
server_key_iv: "6ac087bb4853ca81f7f6bd9dfa4e02e1"
server_key_version: "kuduclusterkey@0"
I20260928 16:52:07.116065 21230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:52:07.142195 21230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:52:07.148383 21230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:52:07.149014 21230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:52:07.150815 21230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:52:07.153013 21230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:52:07.153138 21230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:52:07.153303 21230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:52:07.153378 21230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:52:07.228623 21230 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:40687
I20260928 16:52:07.228641 21357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:40687 every 8 connection(s)
I20260928 16:52:07.230314 21230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:52:07.239243 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 21230
I20260928 16:52:07.239486 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:52:07.240105 9588 external_mini_cluster.cc:1428] Setting key 5a96f0a2df33a6ef1d5f6aa5d6117486
I20260928 16:52:07.247634 21217 heartbeater.cc:503] Master 127.9.93.62:34143 was elected leader, sending a full tablet report...
Sep 28 16:52:07 dist-test-slave-wzcb krb5kdc[20525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614327, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:52:07.258719 21360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:07.233898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:55519 (local address 127.9.93.62:34143)
0928 16:52:07.234800 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0928 16:52:07.234805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:52:07.236200 (+ 1395us) server_negotiation.cc:407] Connection header received
0928 16:52:07.238485 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:52:07.238490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:52:07.238725 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:52:07.238877 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:52:07.241128 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:07.242274 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:07.244447 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:07.245173 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:07.245317 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:07.249805 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:52:07.249812 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:52:07.249817 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:52:07.249859 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:52:07.253421 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:52:07.254685 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:52:07.254690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:52:07.254692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:52:07.254949 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:52:07.255606 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:52:07.255611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:52:07.255614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:52:07.256109 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:52:07.256264 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 16:52:07.257618 (+ 1354us) server_negotiation.cc:299] Negotiation successful
0928 16:52:07.258028 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765,"thread_start_us":633,"threads_started":1}
I20260928 16:52:07.259809 21361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:07.234804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:34143 (local address 127.9.93.2:55519)
0928 16:52:07.236008 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0928 16:52:07.236055 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:52:07.238099 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:52:07.239276 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:52:07.239290 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:52:07.240041 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:52:07.240918 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:52:07.240945 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:52:07.242492 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:52:07.242497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:52:07.244251 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:52:07.244268 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:52:07.245527 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:52:07.245537 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:52:07.245830 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:07.246615 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:52:07.246642 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:52:07.249523 (+ 2881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:52:07.253694 (+ 4171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:52:07.253706 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:52:07.253731 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:52:07.254539 (+ 808us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:52:07.255088 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:52:07.255093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:52:07.255097 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:52:07.255473 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:52:07.256342 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:52:07.256354 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:52:07.257058 (+ 704us) client_negotiation.cc:769] Sending connection context
0928 16:52:07.257398 (+ 340us) client_negotiation.cc:240] Negotiation successful
0928 16:52:07.258056 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"thread_start_us":593,"threads_started":1}
I20260928 16:52:07.261590 21358 heartbeater.cc:347] Connected to a master server at 127.9.93.62:34143
I20260928 16:52:07.261972 21358 heartbeater.cc:464] Registering TS with master...
I20260928 16:52:07.262887 21358 heartbeater.cc:511] Master 127.9.93.62:34143 requested a full tablet report, sending...
I20260928 16:52:07.265096 21001 ts_manager.cc:194] Registered new tserver with Master: 5cb322a7251040d6b7e72f32be6e6f73 (127.9.93.2:40687)
I20260928 16:52:07.266106 21001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.2@KRBTEST.COM'} at 127.9.93.2:55519
WARNING: no policy specified for kudu/127.9.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
I20260928 16:52:07.308581 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.93.20:47663/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:34143
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:52:08.068130 21364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:52:08.068671 21364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:52:08.068893 21364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:52:08.081709 21364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:52:08.081902 21364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:52:08.082104 21364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:52:08.096784 21364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.93.20:47663/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:34143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:52:08.098615 21364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:52:08.100694 21364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:52:08.123343 21369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:52:08.123797 21372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:52:08.124235 21370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:52:08.126420 21364 server_base.cc:1034] running on GCE node
I20260928 16:52:08.127434 21364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:52:08.128801 21364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:52:08.130131 21364 hybrid_clock.cc:648] HybridClock initialized: now 1790614328129981 us; error 69 us; skew 500 ppm
I20260928 16:52:08.130671 21364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:52:08 dist-test-slave-wzcb krb5kdc[20525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614328, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:52:08.136857 21364 init.cc:377] Logged in from keytab as kudu/127.9.93.3@KRBTEST.COM (short username kudu)
I20260928 16:52:08.139711 21364 webserver.cc:460] Webserver started at http://127.9.93.3:36721/ using document root <none> and password file <none>
I20260928 16:52:08.140820 21364 fs_manager.cc:359] Metadata directory not provided
I20260928 16:52:08.140977 21364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:52:08.141338 21364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:52:08 dist-test-slave-wzcb krb5kdc[20525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614328, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for HTTP/127.9.93.20@KRBTEST.COM
I20260928 16:52:08.159022 21364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "9fbd1fe4dafc40bfbbe5e68943b53f62"
format_stamp: "Formatted at 2026-09-28 16:52:08 on dist-test-slave-wzcb"
server_key: "78e187ad26c16542868352455583efe1"
server_key_iv: "655bb55049d7a06819252021de61ada0"
server_key_version: "kuduclusterkey@0"
I20260928 16:52:08.160887 21364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "9fbd1fe4dafc40bfbbe5e68943b53f62"
format_stamp: "Formatted at 2026-09-28 16:52:08 on dist-test-slave-wzcb"
server_key: "78e187ad26c16542868352455583efe1"
server_key_iv: "655bb55049d7a06819252021de61ada0"
server_key_version: "kuduclusterkey@0"
I20260928 16:52:08.175546 21364 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.006s sys 0.006s
I20260928 16:52:08.197821 21379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:52:08.200341 21364 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.001s
I20260928 16:52:08.200721 21364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "9fbd1fe4dafc40bfbbe5e68943b53f62"
format_stamp: "Formatted at 2026-09-28 16:52:08 on dist-test-slave-wzcb"
server_key: "78e187ad26c16542868352455583efe1"
server_key_iv: "655bb55049d7a06819252021de61ada0"
server_key_version: "kuduclusterkey@0"
I20260928 16:52:08.201217 21364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:52:08.230353 21364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:52:08.236791 21364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:52:08.237298 21364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:52:08.239220 21364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:52:08.241605 21364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:52:08.241824 21364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:52:08.242008 21364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:52:08.242091 21364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:52:08.269658 21358 heartbeater.cc:503] Master 127.9.93.62:34143 was elected leader, sending a full tablet report...
I20260928 16:52:08.331379 21364 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:45427
I20260928 16:52:08.331435 21491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:45427 every 8 connection(s)
I20260928 16:52:08.333165 21364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:52:08.338408 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 21364
I20260928 16:52:08.338605 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:52:08.339246 9588 external_mini_cluster.cc:1428] Setting key 52cbad870ceb4f68aca9786f7fa9c5cb
Sep 28 16:52:08 dist-test-slave-wzcb krb5kdc[20525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614328, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:52:08.363163 21494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:08.337278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:35831 (local address 127.9.93.62:34143)
0928 16:52:08.338282 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0928 16:52:08.338289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:52:08.340518 (+ 2229us) server_negotiation.cc:407] Connection header received
0928 16:52:08.343167 (+ 2649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:52:08.343171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:52:08.343341 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:52:08.343463 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:52:08.345973 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.347121 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.349911 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.350761 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.350893 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:08.355677 (+ 4784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:52:08.355682 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:52:08.355686 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:52:08.355726 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:52:08.359350 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:52:08.360437 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:52:08.360442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:52:08.360444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:52:08.360591 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:52:08.361115 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:52:08.361118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:52:08.361120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:52:08.361423 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:52:08.361591 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 16:52:08.362471 (+ 880us) server_negotiation.cc:299] Negotiation successful
0928 16:52:08.362745 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":631,"threads_started":1}
I20260928 16:52:08.364609 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:08.338203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:34143 (local address 127.9.93.3:35831)
0928 16:52:08.340279 (+ 2076us) negotiation.cc:107] Waiting for socket to connect
0928 16:52:08.340337 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 16:52:08.342863 (+ 2526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:52:08.343883 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:52:08.343895 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:52:08.344638 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:52:08.345722 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:52:08.345752 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.347410 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.347416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:52:08.349663 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:52:08.349688 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.351091 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.351099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:52:08.351565 (+ 466us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:08.352421 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:52:08.352447 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:52:08.355358 (+ 2911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:52:08.359556 (+ 4198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:52:08.359565 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:52:08.359583 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:52:08.360292 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:52:08.360687 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:52:08.360691 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:52:08.360693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:52:08.361018 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:52:08.361630 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:52:08.361639 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:52:08.362028 (+ 389us) client_negotiation.cc:769] Sending connection context
0928 16:52:08.362343 (+ 315us) client_negotiation.cc:240] Negotiation successful
0928 16:52:08.363037 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1710,"thread_start_us":1206,"threads_started":1}
I20260928 16:52:08.366497 21492 heartbeater.cc:347] Connected to a master server at 127.9.93.62:34143
I20260928 16:52:08.367072 21492 heartbeater.cc:464] Registering TS with master...
I20260928 16:52:08.368423 21492 heartbeater.cc:511] Master 127.9.93.62:34143 requested a full tablet report, sending...
I20260928 16:52:08.370541 21001 ts_manager.cc:194] Registered new tserver with Master: 9fbd1fe4dafc40bfbbe5e68943b53f62 (127.9.93.3:45427)
I20260928 16:52:08.371640 21001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.3@KRBTEST.COM'} at 127.9.93.3:35831
I20260928 16:52:08.382293 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:52:08.409221 21494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:08.388627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45404 (local address 127.9.93.62:34143)
0928 16:52:08.388867 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 16:52:08.388874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:52:08.389713 (+ 839us) server_negotiation.cc:407] Connection header received
0928 16:52:08.389904 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:52:08.389909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:52:08.390026 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:52:08.390117 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:52:08.392634 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.393887 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.396269 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.396845 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.396980 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:08.400826 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:52:08.400833 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:52:08.400836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:52:08.400884 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:52:08.405214 (+ 4330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:52:08.406616 (+ 1402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:52:08.406622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:52:08.406624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:52:08.406904 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:52:08.407469 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:52:08.407474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:52:08.407477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:52:08.407894 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:52:08.408187 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 16:52:08.408546 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 16:52:08.408817 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 16:52:08.414959 21001 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:52:08.426029 21001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:52:08.463855 21506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:08.445989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:43611 (local address 127.0.0.1:38894)
0928 16:52:08.448411 (+ 2422us) negotiation.cc:107] Waiting for socket to connect
0928 16:52:08.448469 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 16:52:08.448795 (+ 326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:52:08.450483 (+ 1688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:52:08.450494 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:52:08.450535 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:52:08.451650 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:52:08.451668 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.454739 (+ 3071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.454747 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:52:08.458190 (+ 3443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:52:08.458215 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.462033 (+ 3818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.462043 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:52:08.462443 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:08.462548 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 16:52:08.462694 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 16:52:08.462952 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2071,"thread_start_us":1815,"threads_started":1}
I20260928 16:52:08.465765 21505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:08.446073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38894 (local address 127.9.93.1:43611)
0928 16:52:08.447459 (+ 1386us) server_negotiation.cc:206] Beginning negotiation
0928 16:52:08.447470 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:52:08.449032 (+ 1562us) server_negotiation.cc:407] Connection header received
0928 16:52:08.449179 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:52:08.449189 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:52:08.449576 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:52:08.449896 (+ 320us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:52:08.451951 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.454494 (+ 2543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.458491 (+ 3997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.461087 (+ 2596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.461274 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:08.461744 (+ 470us) server_negotiation.cc:1035] Waiting for connection context
0928 16:52:08.463265 (+ 1521us) server_negotiation.cc:299] Negotiation successful
0928 16:52:08.465009 (+ 1744us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1039,"thread_start_us":738,"threads_started":1}
I20260928 16:52:08.472216 21151 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4b9c2625c74702a6337c20e2c1706d (DEFAULT_TABLE table=test-table [id=fbaf37cd5abf44159c819449fc98c230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:52:08.474694 21151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4b9c2625c74702a6337c20e2c1706d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:52:08.475555 21506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:08.449009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:40687 (local address 127.0.0.1:57408)
0928 16:52:08.464481 (+ 15472us) negotiation.cc:107] Waiting for socket to connect
0928 16:52:08.464502 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:52:08.465936 (+ 1434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:52:08.467118 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:52:08.467122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:52:08.467146 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:52:08.467392 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:52:08.467406 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.470368 (+ 2962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.470373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:52:08.472313 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:52:08.472327 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.474242 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.474249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:52:08.474545 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:08.474573 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 16:52:08.474653 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 16:52:08.475057 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15346,"thread_start_us":16545,"threads_started":1}
I20260928 16:52:08.475594 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:08.449734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57408 (local address 127.9.93.2:40687)
0928 16:52:08.455805 (+ 6071us) server_negotiation.cc:206] Beginning negotiation
0928 16:52:08.455817 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:52:08.464634 (+ 8817us) server_negotiation.cc:407] Connection header received
0928 16:52:08.466230 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:52:08.466241 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:52:08.466575 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:52:08.466749 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:52:08.467606 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.470113 (+ 2507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.472535 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.474006 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.474145 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:08.474456 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0928 16:52:08.474751 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 16:52:08.474995 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5822,"thread_start_us":5484,"threads_started":1}
I20260928 16:52:08.478180 21507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:08.446571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:45427 (local address 127.0.0.1:55148)
0928 16:52:08.464191 (+ 17620us) negotiation.cc:107] Waiting for socket to connect
0928 16:52:08.464214 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:52:08.464390 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:52:08.467415 (+ 3025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:52:08.467420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:52:08.467443 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:52:08.468312 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:52:08.468329 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.470982 (+ 2653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.470988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:52:08.474216 (+ 3228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:52:08.474235 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.476962 (+ 2727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.476972 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:52:08.477300 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:08.477335 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:52:08.477430 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 16:52:08.477607 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17444,"thread_start_us":17928,"threads_started":1}
I20260928 16:52:08.478750 21508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:08.447418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55148 (local address 127.9.93.3:45427)
0928 16:52:08.466558 (+ 19140us) server_negotiation.cc:206] Beginning negotiation
0928 16:52:08.466568 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:52:08.466649 (+ 81us) server_negotiation.cc:407] Connection header received
0928 16:52:08.466769 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:52:08.466777 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:52:08.467132 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:52:08.467420 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:52:08.468733 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.470772 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.474465 (+ 3693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.476451 (+ 1986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.476605 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:08.477698 (+ 1093us) server_negotiation.cc:1035] Waiting for connection context
0928 16:52:08.477800 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 16:52:08.478089 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18887,"thread_start_us":18712,"threads_started":1}
I20260928 16:52:08.481066 21293 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4b9c2625c74702a6337c20e2c1706d (DEFAULT_TABLE table=test-table [id=fbaf37cd5abf44159c819449fc98c230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:52:08.483155 21293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4b9c2625c74702a6337c20e2c1706d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:52:08.483412 21427 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4b9c2625c74702a6337c20e2c1706d (DEFAULT_TABLE table=test-table [id=fbaf37cd5abf44159c819449fc98c230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:52:08.485214 21427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4b9c2625c74702a6337c20e2c1706d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:52:08.504752 21512 tablet_bootstrap.cc:492] T 7e4b9c2625c74702a6337c20e2c1706d P cc5e55e26c2b47809f1905424c848bab: Bootstrap starting.
I20260928 16:52:08.507860 21513 tablet_bootstrap.cc:492] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73: Bootstrap starting.
I20260928 16:52:08.510510 21511 tablet_bootstrap.cc:492] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62: Bootstrap starting.
I20260928 16:52:08.514137 21512 tablet_bootstrap.cc:654] T 7e4b9c2625c74702a6337c20e2c1706d P cc5e55e26c2b47809f1905424c848bab: Neither blocks nor log segments found. Creating new log.
I20260928 16:52:08.515898 21513 tablet_bootstrap.cc:654] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73: Neither blocks nor log segments found. Creating new log.
I20260928 16:52:08.516496 21512 log.cc:824] T 7e4b9c2625c74702a6337c20e2c1706d P cc5e55e26c2b47809f1905424c848bab: Log is configured to *not* fsync() on all Append() calls
I20260928 16:52:08.517422 21511 tablet_bootstrap.cc:654] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62: Neither blocks nor log segments found. Creating new log.
I20260928 16:52:08.518349 21513 log.cc:824] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73: Log is configured to *not* fsync() on all Append() calls
I20260928 16:52:08.519997 21511 log.cc:824] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62: Log is configured to *not* fsync() on all Append() calls
I20260928 16:52:08.521984 21512 tablet_bootstrap.cc:492] T 7e4b9c2625c74702a6337c20e2c1706d P cc5e55e26c2b47809f1905424c848bab: No bootstrap required, opened a new log
I20260928 16:52:08.522590 21512 ts_tablet_manager.cc:1397] T 7e4b9c2625c74702a6337c20e2c1706d P cc5e55e26c2b47809f1905424c848bab: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.009s
I20260928 16:52:08.525939 21513 tablet_bootstrap.cc:492] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73: No bootstrap required, opened a new log
I20260928 16:52:08.526324 21513 ts_tablet_manager.cc:1397] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.004s
I20260928 16:52:08.527276 21511 tablet_bootstrap.cc:492] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62: No bootstrap required, opened a new log
I20260928 16:52:08.527714 21511 ts_tablet_manager.cc:1397] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.003s
I20260928 16:52:08.537650 21513 raft_consensus.cc:348] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc5e55e26c2b47809f1905424c848bab" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 43611 } } peers { permanent_uuid: "5cb322a7251040d6b7e72f32be6e6f73" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 40687 } } peers { permanent_uuid: "9fbd1fe4dafc40bfbbe5e68943b53f62" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 45427 } }
I20260928 16:52:08.538301 21513 raft_consensus.cc:374] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:52:08.538463 21513 raft_consensus.cc:729] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb322a7251040d6b7e72f32be6e6f73, State: Initialized, Role: FOLLOWER
I20260928 16:52:08.538769 21512 raft_consensus.cc:348] T 7e4b9c2625c74702a6337c20e2c1706d P cc5e55e26c2b47809f1905424c848bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc5e55e26c2b47809f1905424c848bab" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 43611 } } peers { permanent_uuid: "5cb322a7251040d6b7e72f32be6e6f73" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 40687 } } peers { permanent_uuid: "9fbd1fe4dafc40bfbbe5e68943b53f62" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 45427 } }
I20260928 16:52:08.539486 21513 consensus_queue.cc:260] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc5e55e26c2b47809f1905424c848bab" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 43611 } } peers { permanent_uuid: "5cb322a7251040d6b7e72f32be6e6f73" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 40687 } } peers { permanent_uuid: "9fbd1fe4dafc40bfbbe5e68943b53f62" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 45427 } }
I20260928 16:52:08.539674 21512 raft_consensus.cc:374] T 7e4b9c2625c74702a6337c20e2c1706d P cc5e55e26c2b47809f1905424c848bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:52:08.539851 21512 raft_consensus.cc:729] T 7e4b9c2625c74702a6337c20e2c1706d P cc5e55e26c2b47809f1905424c848bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc5e55e26c2b47809f1905424c848bab, State: Initialized, Role: FOLLOWER
I20260928 16:52:08.540748 21512 consensus_queue.cc:260] T 7e4b9c2625c74702a6337c20e2c1706d P cc5e55e26c2b47809f1905424c848bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc5e55e26c2b47809f1905424c848bab" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 43611 } } peers { permanent_uuid: "5cb322a7251040d6b7e72f32be6e6f73" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 40687 } } peers { permanent_uuid: "9fbd1fe4dafc40bfbbe5e68943b53f62" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 45427 } }
I20260928 16:52:08.542510 21513 ts_tablet_manager.cc:1428] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73: Time spent starting tablet: real 0.016s user 0.008s sys 0.008s
I20260928 16:52:08.543514 21511 raft_consensus.cc:348] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc5e55e26c2b47809f1905424c848bab" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 43611 } } peers { permanent_uuid: "5cb322a7251040d6b7e72f32be6e6f73" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 40687 } } peers { permanent_uuid: "9fbd1fe4dafc40bfbbe5e68943b53f62" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 45427 } }
I20260928 16:52:08.544278 21511 raft_consensus.cc:374] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:52:08.544433 21511 raft_consensus.cc:729] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fbd1fe4dafc40bfbbe5e68943b53f62, State: Initialized, Role: FOLLOWER
I20260928 16:52:08.544874 21512 ts_tablet_manager.cc:1428] T 7e4b9c2625c74702a6337c20e2c1706d P cc5e55e26c2b47809f1905424c848bab: Time spent starting tablet: real 0.022s user 0.020s sys 0.000s
I20260928 16:52:08.545243 21511 consensus_queue.cc:260] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc5e55e26c2b47809f1905424c848bab" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 43611 } } peers { permanent_uuid: "5cb322a7251040d6b7e72f32be6e6f73" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 40687 } } peers { permanent_uuid: "9fbd1fe4dafc40bfbbe5e68943b53f62" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 45427 } }
I20260928 16:52:08.547073 21492 heartbeater.cc:503] Master 127.9.93.62:34143 was elected leader, sending a full tablet report...
I20260928 16:52:08.547960 21511 ts_tablet_manager.cc:1428] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62: Time spent starting tablet: real 0.020s user 0.004s sys 0.016s
W20260928 16:52:08.586113 21493 tablet.cc:2369] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:52:08.653690 21519 raft_consensus.cc:482] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:52:08.654273 21519 raft_consensus.cc:504] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc5e55e26c2b47809f1905424c848bab" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 43611 } } peers { permanent_uuid: "5cb322a7251040d6b7e72f32be6e6f73" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 40687 } } peers { permanent_uuid: "9fbd1fe4dafc40bfbbe5e68943b53f62" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 45427 } }
I20260928 16:52:08.657037 21519 leader_election.cc:288] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc5e55e26c2b47809f1905424c848bab (127.9.93.1:43611), 5cb322a7251040d6b7e72f32be6e6f73 (127.9.93.2:40687)
W20260928 16:52:08.658670 21218 tablet.cc:2369] T 7e4b9c2625c74702a6337c20e2c1706d P cc5e55e26c2b47809f1905424c848bab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:52:08.665400 21517 raft_consensus.cc:482] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:52:08.666821 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:08.657753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:43611 (local address 127.9.93.3:43391)
0928 16:52:08.657975 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 16:52:08.658008 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 16:52:08.658213 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:52:08.658933 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:52:08.658939 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:52:08.658988 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:52:08.659484 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:52:08.659500 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.661275 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.661279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:52:08.663359 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:52:08.663380 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.665568 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.665576 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:52:08.665941 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:08.665986 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 16:52:08.666152 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 16:52:08.666331 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 16:52:08.666893 21505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:08.657840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:43391 (local address 127.9.93.1:43611)
0928 16:52:08.658082 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 16:52:08.658086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:52:08.658110 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:52:08.658449 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:52:08.658456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:52:08.658550 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:52:08.658652 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:52:08.659655 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.661051 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.663663 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.665334 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.665523 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:08.665660 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 16:52:08.666225 (+ 565us) server_negotiation.cc:299] Negotiation successful
0928 16:52:08.666461 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 16:52:08.666313 21517 raft_consensus.cc:504] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc5e55e26c2b47809f1905424c848bab" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 43611 } } peers { permanent_uuid: "5cb322a7251040d6b7e72f32be6e6f73" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 40687 } } peers { permanent_uuid: "9fbd1fe4dafc40bfbbe5e68943b53f62" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 45427 } }
I20260928 16:52:08.669047 21172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4b9c2625c74702a6337c20e2c1706d" candidate_uuid: "9fbd1fe4dafc40bfbbe5e68943b53f62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc5e55e26c2b47809f1905424c848bab" is_pre_election: true
I20260928 16:52:08.670297 21172 raft_consensus.cc:2454] T 7e4b9c2625c74702a6337c20e2c1706d P cc5e55e26c2b47809f1905424c848bab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fbd1fe4dafc40bfbbe5e68943b53f62 in term 0.
I20260928 16:52:08.671571 21520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:08.658253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:40687 (local address 127.9.93.3:56423)
0928 16:52:08.659844 (+ 1591us) negotiation.cc:107] Waiting for socket to connect
0928 16:52:08.659866 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:52:08.660043 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:52:08.660379 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:52:08.660384 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:52:08.660409 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:52:08.661052 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:52:08.661066 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.662792 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.662797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:52:08.665877 (+ 3080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:52:08.665900 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.669564 (+ 3664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.669575 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:52:08.669872 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:08.669921 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 16:52:08.670560 (+ 639us) client_negotiation.cc:240] Negotiation successful
0928 16:52:08.670740 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1401,"thread_start_us":951,"threads_started":1}
I20260928 16:52:08.671705 21517 leader_election.cc:288] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc5e55e26c2b47809f1905424c848bab (127.9.93.1:43611), 9fbd1fe4dafc40bfbbe5e68943b53f62 (127.9.93.3:45427)
I20260928 16:52:08.671610 21380 leader_election.cc:302] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fbd1fe4dafc40bfbbe5e68943b53f62, cc5e55e26c2b47809f1905424c848bab; no voters:
I20260928 16:52:08.671950 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:08.658435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:56423 (local address 127.9.93.2:40687)
0928 16:52:08.658665 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 16:52:08.658671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:52:08.659972 (+ 1301us) server_negotiation.cc:407] Connection header received
0928 16:52:08.660184 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:52:08.660188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:52:08.660254 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:52:08.660341 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:52:08.661265 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.662598 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.666129 (+ 3531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.667986 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.668145 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:08.668274 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 16:52:08.670119 (+ 1845us) server_negotiation.cc:299] Negotiation successful
0928 16:52:08.670297 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 16:52:08.672241 21519 raft_consensus.cc:2781] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:52:08.672583 21519 raft_consensus.cc:482] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:52:08.672820 21519 raft_consensus.cc:3037] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:52:08.678668 21519 raft_consensus.cc:504] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc5e55e26c2b47809f1905424c848bab" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 43611 } } peers { permanent_uuid: "5cb322a7251040d6b7e72f32be6e6f73" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 40687 } } peers { permanent_uuid: "9fbd1fe4dafc40bfbbe5e68943b53f62" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 45427 } }
I20260928 16:52:08.680078 21519 leader_election.cc:288] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62 [CANDIDATE]: Term 1 election: Requested vote from peers cc5e55e26c2b47809f1905424c848bab (127.9.93.1:43611), 5cb322a7251040d6b7e72f32be6e6f73 (127.9.93.2:40687)
I20260928 16:52:08.681495 21172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4b9c2625c74702a6337c20e2c1706d" candidate_uuid: "9fbd1fe4dafc40bfbbe5e68943b53f62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc5e55e26c2b47809f1905424c848bab"
I20260928 16:52:08.681895 21172 raft_consensus.cc:3037] T 7e4b9c2625c74702a6337c20e2c1706d P cc5e55e26c2b47809f1905424c848bab [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:52:08.681991 21313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4b9c2625c74702a6337c20e2c1706d" candidate_uuid: "9fbd1fe4dafc40bfbbe5e68943b53f62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb322a7251040d6b7e72f32be6e6f73" is_pre_election: true
I20260928 16:52:08.682890 21313 raft_consensus.cc:2454] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fbd1fe4dafc40bfbbe5e68943b53f62 in term 0.
I20260928 16:52:08.684628 21313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4b9c2625c74702a6337c20e2c1706d" candidate_uuid: "9fbd1fe4dafc40bfbbe5e68943b53f62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb322a7251040d6b7e72f32be6e6f73"
I20260928 16:52:08.684795 21508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:08.670429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:50039 (local address 127.9.93.3:45427)
0928 16:52:08.670660 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 16:52:08.670667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:52:08.674701 (+ 4034us) server_negotiation.cc:407] Connection header received
0928 16:52:08.674956 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:52:08.674966 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:52:08.675037 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:52:08.675122 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:52:08.676495 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.677829 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.681163 (+ 3334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.683020 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.683146 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:08.683284 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 16:52:08.684131 (+ 847us) server_negotiation.cc:299] Negotiation successful
0928 16:52:08.684307 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:52:08.685117 21313 raft_consensus.cc:3037] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:52:08.687108 21505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:08.669985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:36253 (local address 127.9.93.1:43611)
0928 16:52:08.670272 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 16:52:08.670278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:52:08.674007 (+ 3729us) server_negotiation.cc:407] Connection header received
0928 16:52:08.674452 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:52:08.674459 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:52:08.674608 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:52:08.674685 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:52:08.676618 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.678416 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.682950 (+ 4534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.685254 (+ 2304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.685410 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:08.685617 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 16:52:08.686506 (+ 889us) server_negotiation.cc:299] Negotiation successful
0928 16:52:08.686651 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 16:52:08.688073 21522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:08.670941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:43611 (local address 127.9.93.2:36253)
0928 16:52:08.673832 (+ 2891us) negotiation.cc:107] Waiting for socket to connect
0928 16:52:08.673862 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 16:52:08.674218 (+ 356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:52:08.674969 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:52:08.674974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:52:08.675005 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:52:08.676404 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:52:08.676422 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.678699 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.678705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:52:08.682674 (+ 3969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:52:08.682693 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.686004 (+ 3311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.686014 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:52:08.686268 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:08.686304 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 16:52:08.687351 (+ 1047us) client_negotiation.cc:240] Negotiation successful
0928 16:52:08.687508 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2666,"thread_start_us":3113,"threads_started":1}
I20260928 16:52:08.688546 21172 raft_consensus.cc:2454] T 7e4b9c2625c74702a6337c20e2c1706d P cc5e55e26c2b47809f1905424c848bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fbd1fe4dafc40bfbbe5e68943b53f62 in term 1.
I20260928 16:52:08.689226 21380 leader_election.cc:302] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fbd1fe4dafc40bfbbe5e68943b53f62, cc5e55e26c2b47809f1905424c848bab; no voters:
I20260928 16:52:08.689627 21171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4b9c2625c74702a6337c20e2c1706d" candidate_uuid: "5cb322a7251040d6b7e72f32be6e6f73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc5e55e26c2b47809f1905424c848bab" is_pre_election: true
I20260928 16:52:08.690551 21171 raft_consensus.cc:2379] T 7e4b9c2625c74702a6337c20e2c1706d P cc5e55e26c2b47809f1905424c848bab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5cb322a7251040d6b7e72f32be6e6f73 in current term 1: Already voted for candidate 9fbd1fe4dafc40bfbbe5e68943b53f62 in this term.
I20260928 16:52:08.690670 21313 raft_consensus.cc:2454] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fbd1fe4dafc40bfbbe5e68943b53f62 in term 1.
I20260928 16:52:08.691103 21521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:08.672055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:45427 (local address 127.9.93.2:50039)
0928 16:52:08.674581 (+ 2526us) negotiation.cc:107] Waiting for socket to connect
0928 16:52:08.674597 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:52:08.674773 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:52:08.675165 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:52:08.675169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:52:08.675239 (+ 70us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:52:08.676303 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:52:08.676318 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.678053 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.678059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:52:08.680873 (+ 2814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:52:08.680898 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.683661 (+ 2763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.683669 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:52:08.683896 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:08.683935 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 16:52:08.685088 (+ 1153us) client_negotiation.cc:240] Negotiation successful
0928 16:52:08.685266 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2333,"mutex_wait_us":4,"spinlock_wait_cycles":67968,"thread_start_us":819,"threads_started":1}
I20260928 16:52:08.692427 21519 raft_consensus.cc:2781] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:52:08.693028 21447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4b9c2625c74702a6337c20e2c1706d" candidate_uuid: "5cb322a7251040d6b7e72f32be6e6f73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fbd1fe4dafc40bfbbe5e68943b53f62" is_pre_election: true
I20260928 16:52:08.693797 21519 raft_consensus.cc:686] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62 [term 1 LEADER]: Becoming Leader. State: Replica: 9fbd1fe4dafc40bfbbe5e68943b53f62, State: Running, Role: LEADER
I20260928 16:52:08.694484 21519 consensus_queue.cc:237] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc5e55e26c2b47809f1905424c848bab" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 43611 } } peers { permanent_uuid: "5cb322a7251040d6b7e72f32be6e6f73" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 40687 } } peers { permanent_uuid: "9fbd1fe4dafc40bfbbe5e68943b53f62" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 45427 } }
I20260928 16:52:08.700564 21247 leader_election.cc:302] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cb322a7251040d6b7e72f32be6e6f73; no voters: 9fbd1fe4dafc40bfbbe5e68943b53f62, cc5e55e26c2b47809f1905424c848bab
I20260928 16:52:08.701169 21517 raft_consensus.cc:2726] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 16:52:08.701890 20999 catalog_manager.cc:5495] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fbd1fe4dafc40bfbbe5e68943b53f62 (127.9.93.3). New cstate: current_term: 1 leader_uuid: "9fbd1fe4dafc40bfbbe5e68943b53f62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc5e55e26c2b47809f1905424c848bab" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 43611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb322a7251040d6b7e72f32be6e6f73" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 40687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fbd1fe4dafc40bfbbe5e68943b53f62" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 45427 } health_report { overall_health: HEALTHY } } }
W20260928 16:52:08.733899 21359 tablet.cc:2369] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:52:08.765215 21508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:08.757327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55158 (local address 127.9.93.3:45427)
0928 16:52:08.757541 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 16:52:08.757548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:52:08.757569 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:52:08.757723 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:52:08.757728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:52:08.757820 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:52:08.757983 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:52:08.758540 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.759940 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.761819 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.762686 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.762821 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:08.763641 (+ 820us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:52:08.764277 (+ 636us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:52:08.764432 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 16:52:08.764546 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 16:52:08.764650 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 16:52:08.777647 21313 raft_consensus.cc:1261] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73 [term 1 FOLLOWER]: Refusing update from remote peer 9fbd1fe4dafc40bfbbe5e68943b53f62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:52:08.777933 21171 raft_consensus.cc:1261] T 7e4b9c2625c74702a6337c20e2c1706d P cc5e55e26c2b47809f1905424c848bab [term 1 FOLLOWER]: Refusing update from remote peer 9fbd1fe4dafc40bfbbe5e68943b53f62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:52:08.779062 21523 consensus_queue.cc:1035] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb322a7251040d6b7e72f32be6e6f73" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 40687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260928 16:52:08.779806 21519 consensus_queue.cc:1035] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc5e55e26c2b47809f1905424c848bab" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 43611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:52:08.821543 21000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:52:08.822080 21000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:52:08.827935 21000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b661e3ba844f4311851b718c3f0758b4: Sending DeleteTablet for 3 replicas of tablet 7e4b9c2625c74702a6337c20e2c1706d
I20260928 16:52:08.829802 21151 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e4b9c2625c74702a6337c20e2c1706d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:52:08 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:38894
I20260928 16:52:08.829804 21293 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e4b9c2625c74702a6337c20e2c1706d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:52:08 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:57408
I20260928 16:52:08.830427 21427 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e4b9c2625c74702a6337c20e2c1706d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:52:08 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:55148
I20260928 16:52:08.842059 21537 tablet_replica.cc:330] T 7e4b9c2625c74702a6337c20e2c1706d P cc5e55e26c2b47809f1905424c848bab: stopping tablet replica
I20260928 16:52:08.843330 21537 raft_consensus.cc:2229] T 7e4b9c2625c74702a6337c20e2c1706d P cc5e55e26c2b47809f1905424c848bab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:52:08.844456 21537 raft_consensus.cc:2258] T 7e4b9c2625c74702a6337c20e2c1706d P cc5e55e26c2b47809f1905424c848bab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:52:08.846306 21536 tablet_replica.cc:330] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73: stopping tablet replica
I20260928 16:52:08.846899 21536 raft_consensus.cc:2229] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:52:08.847860 21536 raft_consensus.cc:2258] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:52:08.850510 21536 ts_tablet_manager.cc:1905] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:52:08.850512 21537 ts_tablet_manager.cc:1905] T 7e4b9c2625c74702a6337c20e2c1706d P cc5e55e26c2b47809f1905424c848bab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:52:08.853511 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 21082
I20260928 16:52:08.861466 21536 ts_tablet_manager.cc:1918] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:52:08.861439 21537 ts_tablet_manager.cc:1918] T 7e4b9c2625c74702a6337c20e2c1706d P cc5e55e26c2b47809f1905424c848bab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:52:08.861702 21537 log.cc:1196] T 7e4b9c2625c74702a6337c20e2c1706d P cc5e55e26c2b47809f1905424c848bab: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-0/wal/wals/7e4b9c2625c74702a6337c20e2c1706d
I20260928 16:52:08.862042 21536 log.cc:1196] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/7e4b9c2625c74702a6337c20e2c1706d
I20260928 16:52:08.862847 21538 tablet_replica.cc:330] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62: stopping tablet replica
I20260928 16:52:08.862910 21536 ts_tablet_manager.cc:1939] T 7e4b9c2625c74702a6337c20e2c1706d P 5cb322a7251040d6b7e72f32be6e6f73: Deleting consensus metadata
I20260928 16:52:08.863556 21537 ts_tablet_manager.cc:1939] T 7e4b9c2625c74702a6337c20e2c1706d P cc5e55e26c2b47809f1905424c848bab: Deleting consensus metadata
I20260928 16:52:08.864198 21538 raft_consensus.cc:2229] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:52:08.867735 21538 raft_consensus.cc:2258] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:52:08.869767 20985 catalog_manager.cc:4841] TS cc5e55e26c2b47809f1905424c848bab (127.9.93.1:43611): tablet 7e4b9c2625c74702a6337c20e2c1706d (table test-table [id=fbaf37cd5abf44159c819449fc98c230]) successfully deleted
I20260928 16:52:08.870500 20985 catalog_manager.cc:4841] TS 5cb322a7251040d6b7e72f32be6e6f73 (127.9.93.2:40687): tablet 7e4b9c2625c74702a6337c20e2c1706d (table test-table [id=fbaf37cd5abf44159c819449fc98c230]) successfully deleted
I20260928 16:52:08.873829 21538 ts_tablet_manager.cc:1905] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 16:52:08 dist-test-slave-wzcb krb5kdc[20525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.1@KRBTEST.COM
I20260928 16:52:08.883080 21538 ts_tablet_manager.cc:1918] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:52:08.883507 21538 log.cc:1196] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/7e4b9c2625c74702a6337c20e2c1706d
I20260928 16:52:08.884505 21538 ts_tablet_manager.cc:1939] T 7e4b9c2625c74702a6337c20e2c1706d P 9fbd1fe4dafc40bfbbe5e68943b53f62: Deleting consensus metadata
I20260928 16:52:08.887241 20986 catalog_manager.cc:4841] TS 9fbd1fe4dafc40bfbbe5e68943b53f62 (127.9.93.3:45427): tablet 7e4b9c2625c74702a6337c20e2c1706d (table test-table [id=fbaf37cd5abf44159c819449fc98c230]) successfully deleted
I20260928 16:52:08.894008 21505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:08.855230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38896 (local address 127.9.93.1:43611)
0928 16:52:08.855486 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 16:52:08.855495 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:52:08.864420 (+ 8925us) server_negotiation.cc:407] Connection header received
0928 16:52:08.864643 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:52:08.864648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:52:08.864742 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:52:08.864839 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:52:08.868060 (+ 3221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.869339 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.871711 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:08.872440 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:08.872612 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:08.883407 (+ 10795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:52:08.883440 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:52:08.883468 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:52:08.883543 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:52:08.889228 (+ 5685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:52:08.890686 (+ 1458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:52:08.890697 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:52:08.890699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:52:08.890931 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:52:08.891538 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:52:08.891543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:52:08.891545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:52:08.892140 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:52:08.892549 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0928 16:52:08.893100 (+ 551us) server_negotiation.cc:299] Negotiation successful
0928 16:52:08.893450 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260928 16:52:08.895087 21212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:52:09.057941 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 21082
I20260928 16:52:09.074491 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 21230
Sep 28 16:52:09 dist-test-slave-wzcb krb5kdc[20525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.2@KRBTEST.COM
I20260928 16:52:09.096889 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:09.075419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57424 (local address 127.9.93.2:40687)
0928 16:52:09.075591 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 16:52:09.075595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:52:09.075666 (+ 71us) server_negotiation.cc:407] Connection header received
0928 16:52:09.075877 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:52:09.075881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:52:09.075954 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:52:09.076052 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:52:09.078044 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:09.079016 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:09.081171 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:09.081749 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:09.081878 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:09.087457 (+ 5579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:52:09.087484 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:52:09.087510 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:52:09.087577 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:52:09.092586 (+ 5009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:52:09.093691 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:52:09.093703 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:52:09.093707 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:52:09.094041 (+ 334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:52:09.094620 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:52:09.094626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:52:09.094629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:52:09.095437 (+ 808us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:52:09.095779 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0928 16:52:09.096173 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 16:52:09.096451 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:52:09.097760 21353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:52:09.196394 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 21230
I20260928 16:52:09.221772 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 21364
Sep 28 16:52:09 dist-test-slave-wzcb krb5kdc[20525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.3@KRBTEST.COM
I20260928 16:52:09.242899 21508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:09.222642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55166 (local address 127.9.93.3:45427)
0928 16:52:09.222864 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 16:52:09.222869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:52:09.222961 (+ 92us) server_negotiation.cc:407] Connection header received
0928 16:52:09.223278 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:52:09.223283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:52:09.223423 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:52:09.223561 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:52:09.225603 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:09.226406 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:09.228726 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:09.229205 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:09.229319 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:09.235917 (+ 6598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:52:09.235934 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:52:09.235952 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:52:09.236010 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:52:09.239594 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:52:09.240471 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:52:09.240480 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:52:09.240483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:52:09.240658 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:52:09.241091 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:52:09.241095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:52:09.241097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:52:09.241557 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:52:09.241933 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0928 16:52:09.242246 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 16:52:09.242538 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 16:52:09.243841 21487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:52:09.372810 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 21364
I20260928 16:52:09.392634 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 20969
I20260928 16:52:09.394166 21031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:52:09.511940 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 20969
I20260928 16:52:09.533557 9588 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 16:52:09.870085 9588 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 16:52:09.870366 9588 mini_ranger.cc:66] Stopping Ranger...
I20260928 16:52:10.217296 9588 mini_ranger.cc:68] Stopped Ranger
2026-09-28 16:52:10.217 UTC [20555] LOG: received smart shutdown request
2026-09-28 16:52:10.221 UTC [20555] LOG: background worker "logical replication launcher" (PID 20564) exited with exit code 1
2026-09-28 16:52:10.221 UTC [20559] LOG: shutting down
2026-09-28 16:52:10.406 UTC [20555] LOG: database system is shut down
2026-09-28T16:52:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (59607 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:52:10 dist-test-slave-wzcb krb5kdc[21552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:52:10 dist-test-slave-wzcb krb5kdc[21552](info): set up 2 sockets
Sep 28 16:52:10 dist-test-slave-wzcb krb5kdc[21552](info): commencing operation
krb5kdc: starting...
W20260928 16:52:11.566408 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:52:11 dist-test-slave-wzcb krb5kdc[21552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:52:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:52:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.93.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.93.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.93.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/rangeradmin_127.9.93.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.93.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/rangeradmin_127.9.93.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.93.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.93.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.93.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/rangerlookup_127.9.93.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.93.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/rangerlookup_127.9.93.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.93.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.93.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.93.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/HTTP_127.9.93.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.93.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/HTTP_127.9.93.20@KRBTEST.COM.keytab.
I20260928 16:52:11.946972 9588 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskkzF10S/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/postgres -l logfile start
2026-09-28 16:52:14.890 UTC [21582] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 16:52:14.890 UTC [21582] LOG: listening on IPv4 address "127.9.93.20", port 35115
2026-09-28 16:52:14.894 UTC [21582] LOG: listening on Unix socket "/tmp/.s.PGSQL.35115"
2026-09-28 16:52:14.920 UTC [21585] LOG: database system was shut down at 2026-09-28 16:52:13 UTC
2026-09-28 16:52:14.925 UTC [21582] LOG: database system is ready to accept connections
I20260928 16:52:15.830013 9588 mini_postgres.cc:96] Postgres bound to 35115
2026-09-28 16:52:15.851 UTC [21593] FATAL: database "slave" does not exist
127.9.93.20:35115 - accepting connections
I20260928 16:52:15.851962 9588 mini_ranger.cc:161] Starting Ranger...
I20260928 16:52:15.883924 9588 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 16:52:16.551163 9588 mini_ranger.cc:87] Created ranger Postgres database
I20260928 16:52:16.551280 9588 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ranger-admin
2026-09-28 16:52:16,872 [I] DB FLAVOR :POSTGRES
2026-09-28 16:52:16,872 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 16:52:16,872 [I] Checking connection..
2026-09-28 16:52:16,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:35115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 16:52:17,171 [I] Checking connection passed.
2026-09-28 16:52:17,171 [I] --------- Verifying version history table ---------
2026-09-28 16:52:17,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:35115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:52:17,447 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 16:52:17,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:35115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:52:17,750 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 16:52:17,750 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 16:52:17,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:35115/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 16:52:18.036 UTC [21657] WARNING: there is no transaction in progress
2026-09-28 16:52:18,044 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 16:52:18,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:35115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:52:18,351 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 16:52:18,352 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 16:52:18,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:35115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 16:52:18,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:35115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 16:52:18,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:35115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wzcb.c.gcp-upstream.internal','N') ;"
2026-09-28 16:52:19,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:35115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 16:52:19,493 [I] Table x_portal_user does not exist in database ranger
2026-09-28 16:52:19,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:35115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 16:52:19,762 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 16:52:19,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:35115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 16:52:20,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:35115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 16:52:20,332 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 16:52:20,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:35115/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 16:52:21.462 UTC [21765] WARNING: there is no transaction in progress
2026-09-28 16:52:21.480 UTC [21765] WARNING: there is no transaction in progress
2026-09-28 16:52:21.499 UTC [21765] WARNING: there is no transaction in progress
2026-09-28 16:52:21.517 UTC [21765] WARNING: there is no transaction in progress
2026-09-28 16:52:21.536 UTC [21765] WARNING: there is no transaction in progress
2026-09-28 16:52:21.555 UTC [21765] WARNING: there is no transaction in progress
2026-09-28 16:52:21.619 UTC [21765] WARNING: there is no transaction in progress
2026-09-28 16:52:21.628 UTC [21765] WARNING: there is no transaction in progress
2026-09-28 16:52:21.647 UTC [21765] WARNING: there is no transaction in progress
2026-09-28 16:52:21.659 UTC [21765] WARNING: there is no transaction in progress
2026-09-28 16:52:21.671 UTC [21765] WARNING: there is no transaction in progress
2026-09-28 16:52:21.687 UTC [21765] WARNING: there is no transaction in progress
2026-09-28 16:52:21.699 UTC [21765] WARNING: there is no transaction in progress
2026-09-28 16:52:21.708 UTC [21765] WARNING: there is no transaction in progress
2026-09-28 16:52:21.717 UTC [21765] WARNING: there is no transaction in progress
2026-09-28 16:52:21.725 UTC [21765] WARNING: there is no transaction in progress
2026-09-28 16:52:22,510 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 16:52:22,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:35115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 16:52:22,789 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 16:52:22,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:35115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 16:52:23,072 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 16:52:23,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:35115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 16:52:23,356 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 16:52:23,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:35115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 16:52:23,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:35115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 16:52:23,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:35115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wzcb.c.gcp-upstream.internal';"
2026-09-28 16:52:24,164 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 16:52:24,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:35115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 16:52:24,419 [I] DB_PATCHES have already been applied
I20260928 16:52:24.425776 9588 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ranger-admin:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskkzF10S/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 16:52:24.425886 9588 mini_ranger.cc:193] Using host: 127.9.93.20
I20260928 16:52:24.443042 9588 mini_ranger.cc:227] Ranger admin URL: http://127.9.93.20:54599
Sep 28, 2026 4:52:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:52:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:52:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 4:52:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 4:52:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 4:52:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 4:52:25 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 16:52:25.490869 9588 mini_ranger.cc:160] Time spent starting Ranger: real 9.639s user 0.008s sys 0.045s
Sep 28, 2026 4:52:47 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 4:52:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 4:52:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 4:52:47 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 4:52:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 4:52:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 4:52:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 4:52:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 4:52:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 4:52:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 4:52:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 4:52:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 4:52:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 4:52:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 4:52:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 4:52:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 4:52:47 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 4:52:48 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 16:52:49.401329 9588 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.93.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.93.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.93.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/rangerkms_127.9.93.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.93.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/rangerkms_127.9.93.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.93.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.93.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.93.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/HTTP_127.9.93.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.93.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/HTTP_127.9.93.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 16:52:49 dist-test-slave-wzcb krb5kdc[21552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614369, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 16:52:49.599665 9588 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 16:52:49.627292 9588 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 16:52:50.570391 9588 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 16:52:50.570560 9588 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ranger-kms
I20260928 16:52:50.570612 9588 mini_ranger_kms.cc:227] Using postgres at 127.9.93.20:35115
2026-09-28 16:52:50,956 [I] DB FLAVOR :POSTGRES
2026-09-28 16:52:50,956 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 16:52:50,957 [I] Checking connection
2026-09-28 16:52:50,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:35115/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 16:52:51,248 [I] connection success
2026-09-28 16:52:51,248 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 16:52:51,248 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 16:52:51,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:35115/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 16:52:51,510 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 16:52:51,510 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 16:52:51,510 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 16:52:51,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.93.20:35115/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 16:52:51,790 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 16:52:52.416314 9588 mini_ranger_kms.cc:310] Created kms service
I20260928 16:52:52.642807 9588 mini_ranger_kms.cc:326] Created kudu user
I20260928 16:52:52.700383 9588 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 16:52:52 dist-test-slave-wzcb krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614369, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.93.20@KRBTEST.COM
I20260928 16:52:53.159415 9588 mini_ranger_kms.cc:384] Added ranger policy
I20260928 16:52:53.159976 9588 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ranger-kms:/tmp/dist-test-taskkzF10S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskkzF10S/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 16:52:53.160056 9588 mini_ranger_kms.cc:240] Using host: 127.9.93.20
I20260928 16:52:53.178771 9588 mini_ranger_kms.cc:276] Ranger KMS PID: 21946
I20260928 16:52:53.178918 9588 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.93.20:45181
Sep 28, 2026 4:52:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:52:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:52:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 4:52:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 4:52:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 4:52:54 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 16:52:55.266987 9588 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.667s user 0.064s sys 0.351s
I20260928 16:52:55.267328 9588 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 16:52:55.267546 9588 mini_ranger_kms.cc:401] 127.9.93.20:45181/kms/v1/keys
Sep 28 16:52:55 dist-test-slave-wzcb krb5kdc[21552](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790614375, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.93.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 16:52:55 dist-test-slave-wzcb krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614375, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.93.20@KRBTEST.COM for HTTP/127.9.93.20@KRBTEST.COM
Sep 28 16:52:56 dist-test-slave-wzcb krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614375, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.93.20@KRBTEST.COM for HTTP/127.9.93.20@KRBTEST.COM
Sep 28 16:52:58 dist-test-slave-wzcb krb5kdc[21552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:52:58.923635 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:42761
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:38517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:42761
--ranger_config_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.93.20:45181/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:52:59.314843 21997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:52:59.315212 21997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:52:59.315368 21997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:52:59.323536 21997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:52:59.323681 21997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:52:59.323732 21997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:52:59.323799 21997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:52:59.323853 21997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:52:59.336249 21997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:38517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.93.20:45181/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:42761
--ranger_config_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:42761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:52:59.337783 21997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:52:59.339293 21997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:52:59.353169 22002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:52:59.353197 22005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:52:59.353169 22003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:52:59.353644 21997 server_base.cc:1034] running on GCE node
I20260928 16:52:59.354892 21997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:52:59.356866 21997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:52:59.358139 21997 hybrid_clock.cc:648] HybridClock initialized: now 1790614379358075 us; error 58 us; skew 500 ppm
I20260928 16:52:59.358610 21997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:52:59 dist-test-slave-wzcb krb5kdc[21552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614379, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:52:59.362346 21997 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:52:59.364214 21997 webserver.cc:460] Webserver started at http://127.9.93.62:41347/ using document root <none> and password file <none>
I20260928 16:52:59.365114 21997 fs_manager.cc:359] Metadata directory not provided
I20260928 16:52:59.365245 21997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:52:59.365559 21997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:52:59 dist-test-slave-wzcb krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614379, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for HTTP/127.9.93.20@KRBTEST.COM
I20260928 16:52:59.424304 21997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "6c4a7d424812445f87c874e9c2732c6a"
format_stamp: "Formatted at 2026-09-28 16:52:59 on dist-test-slave-wzcb"
server_key: "e4a78fe2e5b4ebf4f02105d7421a4af8"
server_key_iv: "62903bf9d2f4b4d44d4d401f941aca88"
server_key_version: "kuduclusterkey@0"
I20260928 16:52:59.425673 21997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "6c4a7d424812445f87c874e9c2732c6a"
format_stamp: "Formatted at 2026-09-28 16:52:59 on dist-test-slave-wzcb"
server_key: "e4a78fe2e5b4ebf4f02105d7421a4af8"
server_key_iv: "62903bf9d2f4b4d44d4d401f941aca88"
server_key_version: "kuduclusterkey@0"
I20260928 16:52:59.434304 21997 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260928 16:52:59.455163 22012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:52:59.457346 21997 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260928 16:52:59.457620 21997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "6c4a7d424812445f87c874e9c2732c6a"
format_stamp: "Formatted at 2026-09-28 16:52:59 on dist-test-slave-wzcb"
server_key: "e4a78fe2e5b4ebf4f02105d7421a4af8"
server_key_iv: "62903bf9d2f4b4d44d4d401f941aca88"
server_key_version: "kuduclusterkey@0"
I20260928 16:52:59.457927 21997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:52:59.475523 21997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:52:59.480103 21997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:52:59.480437 21997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:52:59.501070 21997 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:42761
I20260928 16:52:59.501091 22063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:42761 every 8 connection(s)
I20260928 16:52:59.502692 21997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:52:59.507138 22064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:52:59.513087 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 21997
I20260928 16:52:59.513340 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:52:59.514238 9588 external_mini_cluster.cc:1428] Setting key ce8da5c8cf9ec1deda0b2ffd683060d2
I20260928 16:52:59.521369 22064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a: Bootstrap starting.
I20260928 16:52:59.525872 22064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a: Neither blocks nor log segments found. Creating new log.
I20260928 16:52:59.527370 22064 log.cc:824] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:52:59 dist-test-slave-wzcb krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:52:59.531499 22064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a: No bootstrap required, opened a new log
I20260928 16:52:59.540871 22067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:52:59.516713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55252 (local address 127.9.93.62:42761)
0928 16:52:59.517716 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0928 16:52:59.517735 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:52:59.517792 (+ 57us) server_negotiation.cc:407] Connection header received
0928 16:52:59.519934 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:52:59.519967 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:52:59.520487 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:52:59.520907 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:52:59.522817 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:59.524370 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:59.526284 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:52:59.527115 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:52:59.527266 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:52:59.532245 (+ 4979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:52:59.532255 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:52:59.532277 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:52:59.532333 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:52:59.535828 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:52:59.536883 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:52:59.536893 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:52:59.536901 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:52:59.537099 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:52:59.537593 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:52:59.537596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:52:59.537598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:52:59.538310 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:52:59.538615 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 16:52:59.539089 (+ 474us) server_negotiation.cc:299] Negotiation successful
0928 16:52:59.539645 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":442,"threads_started":1}
I20260928 16:52:59.541304 22064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4a7d424812445f87c874e9c2732c6a" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42761 } }
I20260928 16:52:59.541751 22064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:52:59.541874 22064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c4a7d424812445f87c874e9c2732c6a, State: Initialized, Role: FOLLOWER
I20260928 16:52:59.542505 22064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4a7d424812445f87c874e9c2732c6a" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42761 } }
I20260928 16:52:59.542773 22064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:52:59.542930 22064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:52:59.543108 22064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:52:59.545665 22064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4a7d424812445f87c874e9c2732c6a" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42761 } }
I20260928 16:52:59.546289 22064 leader_election.cc:302] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c4a7d424812445f87c874e9c2732c6a; no voters:
I20260928 16:52:59.546996 22064 leader_election.cc:288] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:52:59.547142 22069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:52:59.548339 22069 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [term 1 LEADER]: Becoming Leader. State: Replica: 6c4a7d424812445f87c874e9c2732c6a, State: Running, Role: LEADER
I20260928 16:52:59.548978 22069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4a7d424812445f87c874e9c2732c6a" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42761 } }
I20260928 16:52:59.549700 22064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:52:59.552621 22070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c4a7d424812445f87c874e9c2732c6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4a7d424812445f87c874e9c2732c6a" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42761 } } }
I20260928 16:52:59.553161 22070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [sys.catalog]: This master's current role is: LEADER
I20260928 16:52:59.554185 22071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c4a7d424812445f87c874e9c2732c6a. Latest consensus state: current_term: 1 leader_uuid: "6c4a7d424812445f87c874e9c2732c6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4a7d424812445f87c874e9c2732c6a" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42761 } } }
I20260928 16:52:59.554634 22071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [sys.catalog]: This master's current role is: LEADER
I20260928 16:52:59.556339 22077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:52:59.562434 22077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:52:59.565083 22064 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 16:52:59.584865 22077 catalog_manager.cc:1347] Generated new cluster ID: f0770ff7041f4c919bcc97c07df06448
I20260928 16:52:59.585044 22077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:52:59.615336 22077 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:52:59.617000 22077 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:52:59.625257 22077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a: Generated new TSK 0
I20260928 16:52:59.626952 22077 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
I20260928 16:53:00.460527 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.93.20:45181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:42761
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:38517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:53:00 dist-test-slave-wzcb krb5kdc[21552](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790614380, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 16:53:00.962157 22109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:00.962679 22109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:00.962914 22109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:00.976081 22109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:00.976266 22109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:00.976543 22109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:53:00.994946 22109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:38517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.93.20:45181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:42761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:00.997429 22109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:00.999703 22109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:01.030747 22119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:01.033354 22120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:01.037130 22109 server_base.cc:1034] running on GCE node
W20260928 16:53:01.037590 22122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:01.038904 22109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:01.040684 22109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:01.042047 22109 hybrid_clock.cc:648] HybridClock initialized: now 1790614381041936 us; error 97 us; skew 500 ppm
I20260928 16:53:01.042497 22109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:01 dist-test-slave-wzcb krb5kdc[21552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614381, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:01.046517 22109 init.cc:377] Logged in from keytab as kudu/127.9.93.1@KRBTEST.COM (short username kudu)
I20260928 16:53:01.048671 22109 webserver.cc:460] Webserver started at http://127.9.93.1:39781/ using document root <none> and password file <none>
I20260928 16:53:01.049566 22109 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:01.049715 22109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:53:01.050099 22109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:53:01 dist-test-slave-wzcb krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614381, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for HTTP/127.9.93.20@KRBTEST.COM
I20260928 16:53:01.068262 22109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "d63822f0058c43f6b61ea89c631373c6"
format_stamp: "Formatted at 2026-09-28 16:53:01 on dist-test-slave-wzcb"
server_key: "3d950caf5dcbbc62d0ff834582abe1a0"
server_key_iv: "4cd1496adf12037338d1ded6b660a8a4"
server_key_version: "kuduclusterkey@0"
I20260928 16:53:01.069437 22109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "d63822f0058c43f6b61ea89c631373c6"
format_stamp: "Formatted at 2026-09-28 16:53:01 on dist-test-slave-wzcb"
server_key: "3d950caf5dcbbc62d0ff834582abe1a0"
server_key_iv: "4cd1496adf12037338d1ded6b660a8a4"
server_key_version: "kuduclusterkey@0"
I20260928 16:53:01.076757 22109 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260928 16:53:01.120136 22129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:01.123029 22109 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.004s sys 0.013s
I20260928 16:53:01.123492 22109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "d63822f0058c43f6b61ea89c631373c6"
format_stamp: "Formatted at 2026-09-28 16:53:01 on dist-test-slave-wzcb"
server_key: "3d950caf5dcbbc62d0ff834582abe1a0"
server_key_iv: "4cd1496adf12037338d1ded6b660a8a4"
server_key_version: "kuduclusterkey@0"
I20260928 16:53:01.123966 22109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:01.182077 22109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:01.188685 22109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:01.189163 22109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:01.201761 22109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:53:01.219646 22109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:53:01.219890 22109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:01.220213 22109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:53:01.220465 22109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:53:01 dist-test-slave-wzcb krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614380, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for HTTP/127.9.93.20@KRBTEST.COM
I20260928 16:53:01.541086 22109 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:34831
I20260928 16:53:01.541086 22241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:34831 every 8 connection(s)
I20260928 16:53:01.542937 22109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:53:01.552630 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 22109
I20260928 16:53:01.552915 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:53:01.553813 9588 external_mini_cluster.cc:1428] Setting key 17bf268577e19648fad5a96fa881cb8a
Sep 28 16:53:01 dist-test-slave-wzcb krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614381, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:53:01.584090 22245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:01.547626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42761 (local address 127.9.93.1:44363)
0928 16:53:01.555335 (+ 7709us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:01.555404 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:01.558319 (+ 2915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:01.560600 (+ 2281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:01.560617 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:01.561415 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:53:01.562458 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:01.562494 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:01.564507 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:01.564514 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:01.567079 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:01.567108 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:01.568839 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:01.568847 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:01.569204 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:01.570476 (+ 1272us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:53:01.570515 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:53:01.573688 (+ 3173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:53:01.578391 (+ 4703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:01.578404 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:01.578430 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:01.579152 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:01.579744 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:01.579749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:01.579753 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:01.580059 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:01.580813 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:53:01.580823 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:53:01.581368 (+ 545us) client_negotiation.cc:769] Sending connection context
0928 16:53:01.581722 (+ 354us) client_negotiation.cc:240] Negotiation successful
0928 16:53:01.582436 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7272,"thread_start_us":8093,"threads_started":1}
I20260928 16:53:01.585363 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:01.546736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:44363 (local address 127.9.93.62:42761)
0928 16:53:01.559827 (+ 13091us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:01.559833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:01.559853 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:53:01.559913 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:01.559918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:01.560020 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:01.562697 (+ 2677us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:01.562893 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:01.564155 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:01.567777 (+ 3622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:01.568502 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:01.574084 (+ 5582us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:01.574394 (+ 310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:01.574401 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:01.574406 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:01.574453 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:01.578123 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:01.579406 (+ 1283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:01.579413 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:01.579416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:01.579593 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:01.580221 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:01.580226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:01.580229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:01.580650 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:01.584446 (+ 3796us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:01.584648 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 16:53:01.584959 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12945,"thread_start_us":12778,"threads_started":1}
I20260928 16:53:01.588167 22242 heartbeater.cc:347] Connected to a master server at 127.9.93.62:42761
I20260928 16:53:01.588558 22242 heartbeater.cc:464] Registering TS with master...
I20260928 16:53:01.589622 22242 heartbeater.cc:511] Master 127.9.93.62:42761 requested a full tablet report, sending...
I20260928 16:53:01.592880 22029 ts_manager.cc:194] Registered new tserver with Master: d63822f0058c43f6b61ea89c631373c6 (127.9.93.1:34831)
I20260928 16:53:01.595263 22029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.1@KRBTEST.COM'} at 127.9.93.1:44363
WARNING: no policy specified for kudu/127.9.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
I20260928 16:53:01.643828 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.93.20:45181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:42761
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:38517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:53:01 dist-test-slave-wzcb krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614380, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for HTTP/127.9.93.20@KRBTEST.COM
W20260928 16:53:02.176666 22248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:02.177037 22248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:02.177187 22248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:02.185344 22248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:02.185520 22248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:02.185691 22248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:53:02.196933 22248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:38517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.93.20:45181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:42761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:02.198576 22248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:02.200335 22248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:02.217465 22263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:02.217634 22262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:02.217840 22265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:02.218853 22248 server_base.cc:1034] running on GCE node
I20260928 16:53:02.219853 22248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:02.221383 22248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:02.222641 22248 hybrid_clock.cc:648] HybridClock initialized: now 1790614382222563 us; error 45 us; skew 500 ppm
I20260928 16:53:02.223105 22248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:02 dist-test-slave-wzcb krb5kdc[21552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614382, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:02.226894 22248 init.cc:377] Logged in from keytab as kudu/127.9.93.2@KRBTEST.COM (short username kudu)
I20260928 16:53:02.228822 22248 webserver.cc:460] Webserver started at http://127.9.93.2:46647/ using document root <none> and password file <none>
I20260928 16:53:02.229658 22248 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:02.229800 22248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:53:02.230154 22248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:53:02 dist-test-slave-wzcb krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614382, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for HTTP/127.9.93.20@KRBTEST.COM
I20260928 16:53:02.252035 22248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "aa38467beec143adacf51cfc98e694fc"
format_stamp: "Formatted at 2026-09-28 16:53:02 on dist-test-slave-wzcb"
server_key: "4c875820904cf70defc6a7e9681d2a1d"
server_key_iv: "664e4f3467dfc9a56f6b8b2271d30d1c"
server_key_version: "kuduclusterkey@0"
I20260928 16:53:02.253376 22248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "aa38467beec143adacf51cfc98e694fc"
format_stamp: "Formatted at 2026-09-28 16:53:02 on dist-test-slave-wzcb"
server_key: "4c875820904cf70defc6a7e9681d2a1d"
server_key_iv: "664e4f3467dfc9a56f6b8b2271d30d1c"
server_key_version: "kuduclusterkey@0"
I20260928 16:53:02.261556 22248 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.000s sys 0.008s
I20260928 16:53:02.284175 22272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:02.286674 22248 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.005s
I20260928 16:53:02.286937 22248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "aa38467beec143adacf51cfc98e694fc"
format_stamp: "Formatted at 2026-09-28 16:53:02 on dist-test-slave-wzcb"
server_key: "4c875820904cf70defc6a7e9681d2a1d"
server_key_iv: "664e4f3467dfc9a56f6b8b2271d30d1c"
server_key_version: "kuduclusterkey@0"
I20260928 16:53:02.287261 22248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:02.323578 22248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:02.329536 22248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:02.329927 22248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:02.331583 22248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:53:02.334194 22248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:53:02.334321 22248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:02.334514 22248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:53:02.334580 22248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:02.382951 22248 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:45165
I20260928 16:53:02.383008 22384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:45165 every 8 connection(s)
I20260928 16:53:02.385013 22248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:53:02.393726 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 22248
I20260928 16:53:02.393945 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:53:02.394588 9588 external_mini_cluster.cc:1428] Setting key 66ad720aba66dd27c5ec8dc342370037
Sep 28 16:53:02 dist-test-slave-wzcb krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614382, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.3@KRBTEST.COM" created.
I20260928 16:53:02.433023 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:02.389392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42761 (local address 127.9.93.2:44771)
0928 16:53:02.400277 (+ 10885us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:02.400359 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:02.402892 (+ 2533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:02.408832 (+ 5940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:02.408850 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:02.410603 (+ 1753us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:53:02.411688 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:02.411728 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:02.413644 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:02.413649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:02.416024 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:02.416050 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:02.417559 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:02.417567 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:02.417920 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:02.418894 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:53:02.418934 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:53:02.421990 (+ 3056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:53:02.426524 (+ 4534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:02.426536 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:02.426560 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:02.427587 (+ 1027us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:02.428212 (+ 625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:02.428217 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:02.428221 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:02.428699 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:02.429616 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:53:02.429626 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:53:02.430213 (+ 587us) client_negotiation.cc:769] Sending connection context
0928 16:53:02.430574 (+ 361us) client_negotiation.cc:240] Negotiation successful
0928 16:53:02.431232 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10495,"thread_start_us":10197,"threads_started":1}
I20260928 16:53:02.435261 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:02.388644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:44771 (local address 127.9.93.62:42761)
0928 16:53:02.407904 (+ 19260us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:02.407912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:02.407944 (+ 32us) server_negotiation.cc:407] Connection header received
0928 16:53:02.408035 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:02.408042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:02.408147 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:02.411998 (+ 3851us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:02.412224 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:02.413374 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:02.416320 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:02.417007 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:02.417154 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:02.422390 (+ 5236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:02.422399 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:02.422404 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:02.422466 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:02.426259 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:02.427809 (+ 1550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:02.427816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:02.427827 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:02.428064 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:02.428866 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:02.428871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:02.428875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:02.429464 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:02.429658 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:02.434521 (+ 4863us) server_negotiation.cc:299] Negotiation successful
0928 16:53:02.434842 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19078,"thread_start_us":18901,"threads_started":1}
I20260928 16:53:02.437304 22385 heartbeater.cc:347] Connected to a master server at 127.9.93.62:42761
I20260928 16:53:02.437714 22385 heartbeater.cc:464] Registering TS with master...
I20260928 16:53:02.438725 22385 heartbeater.cc:511] Master 127.9.93.62:42761 requested a full tablet report, sending...
I20260928 16:53:02.441202 22029 ts_manager.cc:194] Registered new tserver with Master: aa38467beec143adacf51cfc98e694fc (127.9.93.2:45165)
I20260928 16:53:02.442437 22029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.2@KRBTEST.COM'} at 127.9.93.2:44771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
I20260928 16:53:02.468333 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.93.20:45181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:42761
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:38517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:53:02.600481 22242 heartbeater.cc:503] Master 127.9.93.62:42761 was elected leader, sending a full tablet report...
W20260928 16:53:02.895987 22391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:02.896333 22391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:02.896456 22391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:02.904806 22391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:02.904955 22391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:02.905097 22391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:53:02.916905 22391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:38517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.93.20:45181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:42761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:02.918486 22391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:02.920054 22391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:02.936769 22396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:02.937014 22399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:02.936769 22397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:02.938150 22391 server_base.cc:1034] running on GCE node
I20260928 16:53:02.938920 22391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:02.940235 22391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:02.941485 22391 hybrid_clock.cc:648] HybridClock initialized: now 1790614382941414 us; error 65 us; skew 500 ppm
I20260928 16:53:02.941892 22391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:02 dist-test-slave-wzcb krb5kdc[21552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614382, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:02.945779 22391 init.cc:377] Logged in from keytab as kudu/127.9.93.3@KRBTEST.COM (short username kudu)
I20260928 16:53:02.947824 22391 webserver.cc:460] Webserver started at http://127.9.93.3:37593/ using document root <none> and password file <none>
I20260928 16:53:02.948693 22391 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:02.948844 22391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:53:02.949210 22391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:53:02 dist-test-slave-wzcb krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614382, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for HTTP/127.9.93.20@KRBTEST.COM
I20260928 16:53:02.966995 22391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "83c97d77def145cea0d9d317211e7b88"
format_stamp: "Formatted at 2026-09-28 16:53:02 on dist-test-slave-wzcb"
server_key: "374649a12dec53a97303fefa59434876"
server_key_iv: "f93f0d6ac4ccb9b24f52abc05d857522"
server_key_version: "kuduclusterkey@0"
I20260928 16:53:02.968091 22391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "83c97d77def145cea0d9d317211e7b88"
format_stamp: "Formatted at 2026-09-28 16:53:02 on dist-test-slave-wzcb"
server_key: "374649a12dec53a97303fefa59434876"
server_key_iv: "f93f0d6ac4ccb9b24f52abc05d857522"
server_key_version: "kuduclusterkey@0"
I20260928 16:53:02.974468 22391 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:53:02.992704 22406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:02.995086 22391 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:53:02.995443 22391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "83c97d77def145cea0d9d317211e7b88"
format_stamp: "Formatted at 2026-09-28 16:53:02 on dist-test-slave-wzcb"
server_key: "374649a12dec53a97303fefa59434876"
server_key_iv: "f93f0d6ac4ccb9b24f52abc05d857522"
server_key_version: "kuduclusterkey@0"
I20260928 16:53:02.995779 22391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:03.014952 22391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:03.019906 22391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:03.020231 22391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:03.021422 22391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:53:03.023021 22391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:53:03.023103 22391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:03.023206 22391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:53:03.023255 22391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:03.059257 22391 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:36935
I20260928 16:53:03.059320 22518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:36935 every 8 connection(s)
I20260928 16:53:03.060886 22391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:53:03.069473 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 22391
I20260928 16:53:03.069645 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:53:03.070184 9588 external_mini_cluster.cc:1428] Setting key 1d6c638b07c679835929d4d07369625c
Sep 28 16:53:03 dist-test-slave-wzcb krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614382, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:53:03.084689 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:03.064089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:37097 (local address 127.9.93.62:42761)
0928 16:53:03.064659 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:03.064665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:03.065869 (+ 1204us) server_negotiation.cc:407] Connection header received
0928 16:53:03.068029 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:03.068036 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:03.068134 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:03.068242 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:03.070079 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:03.070939 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:03.073072 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:03.073704 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:03.073847 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:03.077809 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:03.077815 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:03.077819 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:03.077871 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:03.080919 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:03.081891 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:03.081895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:03.081898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:03.082063 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:03.082635 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:03.082639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:03.082642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:03.082985 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:03.083111 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:03.084025 (+ 914us) server_negotiation.cc:299] Negotiation successful
0928 16:53:03.084335 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":276,"threads_started":1}
I20260928 16:53:03.085901 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:03.064719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42761 (local address 127.9.93.3:37097)
0928 16:53:03.065694 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:03.065739 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:03.067694 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:03.068554 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:03.068564 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:03.069071 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:53:03.069856 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:03.069883 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:03.071170 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:03.071175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:03.072899 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:03.072916 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:03.073957 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:03.073963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:03.074230 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:03.074967 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:53:03.074992 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:53:03.077498 (+ 2506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:53:03.081130 (+ 3632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:03.081140 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:03.081160 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:03.081756 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:03.082183 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:03.082187 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:03.082189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:03.082508 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:03.083122 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:53:03.083130 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:53:03.083618 (+ 488us) client_negotiation.cc:769] Sending connection context
0928 16:53:03.083890 (+ 272us) client_negotiation.cc:240] Negotiation successful
0928 16:53:03.084394 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":428,"threads_started":1}
I20260928 16:53:03.087502 22519 heartbeater.cc:347] Connected to a master server at 127.9.93.62:42761
I20260928 16:53:03.087849 22519 heartbeater.cc:464] Registering TS with master...
I20260928 16:53:03.088696 22519 heartbeater.cc:511] Master 127.9.93.62:42761 requested a full tablet report, sending...
I20260928 16:53:03.090687 22029 ts_manager.cc:194] Registered new tserver with Master: 83c97d77def145cea0d9d317211e7b88 (127.9.93.3:36935)
I20260928 16:53:03.091675 22029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.3@KRBTEST.COM'} at 127.9.93.3:37097
I20260928 16:53:03.099797 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:53:03.101715 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 22109
Sep 28 16:53:03 dist-test-slave-wzcb krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.1@KRBTEST.COM
I20260928 16:53:03.121234 22524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:03.103010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40750 (local address 127.9.93.1:34831)
0928 16:53:03.103694 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:03.103702 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:03.103737 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:53:03.103819 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:03.103826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:03.104106 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:03.104262 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:03.105998 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:03.107219 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:03.109047 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:03.109806 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:03.109951 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:03.114812 (+ 4861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:03.114832 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:03.114851 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:03.114906 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:03.118090 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:03.118946 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:03.118955 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:03.118957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:03.119125 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:03.119530 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:03.119535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:03.119537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:03.120026 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:03.120323 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:03.120636 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 16:53:03.120911 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":335,"threads_started":1}
I20260928 16:53:03.122741 22237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:03.214018 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 22109
I20260928 16:53:03.228581 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 22248
Sep 28 16:53:03 dist-test-slave-wzcb krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.2@KRBTEST.COM
I20260928 16:53:03.252058 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:03.229534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43238 (local address 127.9.93.2:45165)
0928 16:53:03.230180 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:03.230189 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:03.230219 (+ 30us) server_negotiation.cc:407] Connection header received
0928 16:53:03.230304 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:03.230311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:03.230558 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:03.230710 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:03.232173 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:03.233309 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:03.234818 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:03.235607 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:03.235721 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:03.245317 (+ 9596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:03.245339 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:03.245358 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:03.245412 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:03.248738 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:03.249745 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:03.249755 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:03.249757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:03.249941 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:03.250340 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:03.250343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:03.250346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:03.250816 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:03.251083 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:03.251445 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 16:53:03.251746 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":305,"threads_started":1}
I20260928 16:53:03.253489 22380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:03.342576 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 22248
I20260928 16:53:03.357436 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 22391
Sep 28 16:53:03 dist-test-slave-wzcb krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.3@KRBTEST.COM
I20260928 16:53:03.377550 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:03.358621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51846 (local address 127.9.93.3:36935)
0928 16:53:03.359324 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:03.359336 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:03.359375 (+ 39us) server_negotiation.cc:407] Connection header received
0928 16:53:03.359453 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:03.359460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:03.359731 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:03.359914 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:03.361417 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:03.362514 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:03.364220 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:03.364892 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:03.365057 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:03.370393 (+ 5336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:03.370412 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:03.370430 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:03.370492 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:03.373848 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:03.374992 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:03.375001 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:03.375004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:03.375191 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:03.375710 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:03.375714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:03.375716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:03.376156 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:03.376451 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:03.376850 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 16:53:03.377138 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":339,"threads_started":1}
I20260928 16:53:03.379168 22514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:03.467804 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 22391
I20260928 16:53:03.481846 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 21997
I20260928 16:53:03.483500 22059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:03.605113 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 21997
I20260928 16:53:03.626704 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:42761
--webserver_interface=127.9.93.62
--webserver_port=41347
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:38517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:42761
--ranger_config_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.93.20:45181/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.93.20:45181/kms with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:53:04.003773 22531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:04.004195 22531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:04.004374 22531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:04.013310 22531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:53:04.013499 22531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:04.013590 22531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:04.013679 22531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:53:04.013749 22531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:53:04.026525 22531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:38517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.93.20:45181/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:42761
--ranger_config_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:42761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=41347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:04.028241 22531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:04.030043 22531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:04.043828 22537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:04.043912 22539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:04.043946 22536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:04.044970 22531 server_base.cc:1034] running on GCE node
I20260928 16:53:04.045965 22531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:04.047818 22531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:04.049064 22531 hybrid_clock.cc:648] HybridClock initialized: now 1790614384048993 us; error 59 us; skew 500 ppm
I20260928 16:53:04.049503 22531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:04 dist-test-slave-wzcb krb5kdc[21552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614384, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:04.058017 22531 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:53:04.060218 22531 webserver.cc:460] Webserver started at http://127.9.93.62:41347/ using document root <none> and password file <none>
I20260928 16:53:04.061067 22531 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:04.061225 22531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 16:53:04 dist-test-slave-wzcb krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614384, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for HTTP/127.9.93.20@KRBTEST.COM
I20260928 16:53:04.118713 22531 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:53:04.123023 22547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:04.125027 22531 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260928 16:53:04.125293 22531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "6c4a7d424812445f87c874e9c2732c6a"
format_stamp: "Formatted at 2026-09-28 16:52:59 on dist-test-slave-wzcb"
server_key: "e4a78fe2e5b4ebf4f02105d7421a4af8"
server_key_iv: "62903bf9d2f4b4d44d4d401f941aca88"
server_key_version: "kuduclusterkey@0"
I20260928 16:53:04.126616 22531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:04.162988 22531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:04.167901 22531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:04.168233 22531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:04.189666 22531 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:42761
I20260928 16:53:04.189697 22598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:42761 every 8 connection(s)
I20260928 16:53:04.191525 22531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:53:04.195382 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 22531
I20260928 16:53:04.196287 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.93.20:45181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:34831
--local_ip_for_outbound_sockets=127.9.93.1
--tserver_master_addrs=127.9.93.62:42761
--webserver_port=39781
--webserver_interface=127.9.93.1
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:38517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.93.20:45181/kms with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:53:04.199954 22599 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:53:04.203986 22599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a: Bootstrap starting.
I20260928 16:53:04.226902 22599 log.cc:824] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a: Log is configured to *not* fsync() on all Append() calls
I20260928 16:53:04.238301 22599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:53:04.238888 22599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a: Bootstrap complete.
I20260928 16:53:04.248782 22599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4a7d424812445f87c874e9c2732c6a" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42761 } }
I20260928 16:53:04.249919 22599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c4a7d424812445f87c874e9c2732c6a, State: Initialized, Role: FOLLOWER
I20260928 16:53:04.250702 22599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4a7d424812445f87c874e9c2732c6a" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42761 } }
I20260928 16:53:04.250968 22599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:53:04.251096 22599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:53:04.251272 22599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:53:04.256006 22599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4a7d424812445f87c874e9c2732c6a" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42761 } }
I20260928 16:53:04.256606 22599 leader_election.cc:302] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c4a7d424812445f87c874e9c2732c6a; no voters:
I20260928 16:53:04.257323 22599 leader_election.cc:288] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 16:53:04.257527 22603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [term 2 FOLLOWER]: Leader election won for term 2
I20260928 16:53:04.258638 22603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [term 2 LEADER]: Becoming Leader. State: Replica: 6c4a7d424812445f87c874e9c2732c6a, State: Running, Role: LEADER
I20260928 16:53:04.259423 22603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4a7d424812445f87c874e9c2732c6a" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42761 } }
I20260928 16:53:04.259966 22599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:53:04.262275 22605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c4a7d424812445f87c874e9c2732c6a. Latest consensus state: current_term: 2 leader_uuid: "6c4a7d424812445f87c874e9c2732c6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4a7d424812445f87c874e9c2732c6a" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42761 } } }
I20260928 16:53:04.262789 22605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [sys.catalog]: This master's current role is: LEADER
I20260928 16:53:04.263413 22604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6c4a7d424812445f87c874e9c2732c6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4a7d424812445f87c874e9c2732c6a" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42761 } } }
I20260928 16:53:04.263723 22604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a [sys.catalog]: This master's current role is: LEADER
I20260928 16:53:04.267580 22609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:53:04.275012 22609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:53:04.277016 22609 catalog_manager.cc:1259] Loaded cluster ID: f0770ff7041f4c919bcc97c07df06448
I20260928 16:53:04.277268 22609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:53:04.282178 22609 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:53:04.284310 22609 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a: Loaded TSK: 0
I20260928 16:53:04.286311 22609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:53:04.301445 22599 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260928 16:53:04.713594 22601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:04.714200 22601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:04.714445 22601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:04.724577 22601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:04.724746 22601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:04.725010 22601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:53:04.742013 22601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:38517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.93.20:45181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:34831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=39781
--tserver_master_addrs=127.9.93.62:42761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:04.744313 22601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:04.746482 22601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:04.771241 22637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:04.778599 22638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:04.787497 22640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:04.793685 22601 server_base.cc:1034] running on GCE node
I20260928 16:53:04.794775 22601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:04.796556 22601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:04.800160 22601 hybrid_clock.cc:648] HybridClock initialized: now 1790614384800045 us; error 80 us; skew 500 ppm
I20260928 16:53:04.800751 22601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:04 dist-test-slave-wzcb krb5kdc[21552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614384, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:04.806149 22601 init.cc:377] Logged in from keytab as kudu/127.9.93.1@KRBTEST.COM (short username kudu)
I20260928 16:53:04.812968 22601 webserver.cc:460] Webserver started at http://127.9.93.1:39781/ using document root <none> and password file <none>
I20260928 16:53:04.814348 22601 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:04.814627 22601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 16:53:04 dist-test-slave-wzcb krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614384, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for HTTP/127.9.93.20@KRBTEST.COM
I20260928 16:53:04.855187 22601 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.002s sys 0.005s
I20260928 16:53:04.860004 22648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:04.862444 22601 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260928 16:53:04.862732 22601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "d63822f0058c43f6b61ea89c631373c6"
format_stamp: "Formatted at 2026-09-28 16:53:01 on dist-test-slave-wzcb"
server_key: "3d950caf5dcbbc62d0ff834582abe1a0"
server_key_iv: "4cd1496adf12037338d1ded6b660a8a4"
server_key_version: "kuduclusterkey@0"
I20260928 16:53:04.864492 22601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:04.921296 22601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:04.926296 22601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:04.926653 22601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:04.927947 22601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:53:04.932200 22601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:53:04.932452 22601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 16:53:04.932601 22601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:53:04.932695 22601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:04.989702 22601 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:34831
I20260928 16:53:04.989800 22761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:34831 every 8 connection(s)
I20260928 16:53:04.992126 22601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:53:04.996335 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 22601
I20260928 16:53:04.997138 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.93.20:45181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:45165
--local_ip_for_outbound_sockets=127.9.93.2
--tserver_master_addrs=127.9.93.62:42761
--webserver_port=46647
--webserver_interface=127.9.93.2
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:38517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.93.20:45181/kms with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:53:05 dist-test-slave-wzcb krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614384, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:53:05.057369 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:04.996004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:35107 (local address 127.9.93.62:42761)
0928 16:53:05.020042 (+ 24038us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:05.020069 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:05.020156 (+ 87us) server_negotiation.cc:407] Connection header received
0928 16:53:05.021746 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:05.021776 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:05.022452 (+ 676us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:05.023037 (+ 585us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:05.025610 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:05.027779 (+ 2169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:05.030778 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:05.032100 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:05.032375 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:05.042749 (+ 10374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:05.042771 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:05.042800 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:05.042879 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:05.047431 (+ 4552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:05.049189 (+ 1758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:05.049203 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:05.049214 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:05.049508 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:05.050291 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:05.050298 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:05.050302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:05.051148 (+ 846us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:05.051392 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:05.055263 (+ 3871us) server_negotiation.cc:299] Negotiation successful
0928 16:53:05.055900 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23658,"thread_start_us":24021,"threads_started":1}
I20260928 16:53:05.058143 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:04.996215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42761 (local address 127.9.93.1:35107)
0928 16:53:05.017201 (+ 20986us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:05.017285 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:05.020693 (+ 3408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:05.023351 (+ 2658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:05.023377 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:05.024185 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:53:05.025249 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:05.025290 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:05.028101 (+ 2811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:05.028108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:05.030445 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:05.030477 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:05.032504 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:05.032520 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:05.032908 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:05.037554 (+ 4646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:53:05.037610 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:53:05.042226 (+ 4616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:53:05.047809 (+ 5583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:05.047825 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:05.047850 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:05.048896 (+ 1046us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:05.049697 (+ 801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:05.049704 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:05.049708 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:05.050126 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:05.051512 (+ 1386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:53:05.051523 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:53:05.052089 (+ 566us) client_negotiation.cc:769] Sending connection context
0928 16:53:05.052441 (+ 352us) client_negotiation.cc:240] Negotiation successful
0928 16:53:05.053187 (+ 746us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20539,"thread_start_us":20280,"threads_started":1}
I20260928 16:53:05.062718 22762 heartbeater.cc:347] Connected to a master server at 127.9.93.62:42761
I20260928 16:53:05.063210 22762 heartbeater.cc:464] Registering TS with master...
I20260928 16:53:05.064599 22762 heartbeater.cc:511] Master 127.9.93.62:42761 requested a full tablet report, sending...
W20260928 16:53:05.069308 22762 heartbeater.cc:644] Failed to heartbeat to 127.9.93.62:42761 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 16:53:05.069787 22762 heartbeater.cc:464] Registering TS with master...
I20260928 16:53:05.070261 22762 heartbeater.cc:511] Master 127.9.93.62:42761 requested a full tablet report, sending...
I20260928 16:53:05.072461 22762 heartbeater.cc:464] Registering TS with master...
I20260928 16:53:05.072896 22762 heartbeater.cc:511] Master 127.9.93.62:42761 requested a full tablet report, sending...
W20260928 16:53:05.074708 22762 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:53:05.500837 22766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:05.501446 22766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:05.501681 22766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:05.515763 22766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:05.515969 22766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:05.516183 22766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:53:05.534440 22766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:38517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.93.20:45181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:45165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=46647
--tserver_master_addrs=127.9.93.62:42761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:05.536748 22766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:05.539099 22766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:05.558488 22781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:05.560251 22766 server_base.cc:1034] running on GCE node
W20260928 16:53:05.560369 22780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:05.560761 22783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:05.561942 22766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:05.563550 22766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:05.565048 22766 hybrid_clock.cc:648] HybridClock initialized: now 1790614385564922 us; error 104 us; skew 500 ppm
I20260928 16:53:05.565587 22766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:05 dist-test-slave-wzcb krb5kdc[21552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614385, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:05.570976 22766 init.cc:377] Logged in from keytab as kudu/127.9.93.2@KRBTEST.COM (short username kudu)
I20260928 16:53:05.596376 22766 webserver.cc:460] Webserver started at http://127.9.93.2:46647/ using document root <none> and password file <none>
I20260928 16:53:05.597671 22766 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:05.597946 22766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 16:53:05 dist-test-slave-wzcb krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614385, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for HTTP/127.9.93.20@KRBTEST.COM
I20260928 16:53:05.662495 22766 fs_manager.cc:709] Time spent opening directory manager: real 0.022s user 0.007s sys 0.012s
I20260928 16:53:05.684232 22793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:05.687816 22766 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.012s sys 0.010s
I20260928 16:53:05.688238 22766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "aa38467beec143adacf51cfc98e694fc"
format_stamp: "Formatted at 2026-09-28 16:53:02 on dist-test-slave-wzcb"
server_key: "4c875820904cf70defc6a7e9681d2a1d"
server_key_iv: "664e4f3467dfc9a56f6b8b2271d30d1c"
server_key_version: "kuduclusterkey@0"
I20260928 16:53:05.690822 22766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 28 16:53:05 dist-test-slave-wzcb krb5kdc[21552](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790614385, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:05.746969 22766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:05.754055 22766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:05.754541 22766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:05.763187 22766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:53:05.772250 22766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:53:05.772519 22766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 16:53:05.772714 22766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:53:05.772836 22766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:05.890390 22766 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:45165
I20260928 16:53:05.890437 22907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:45165 every 8 connection(s)
I20260928 16:53:05.892946 22766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:53:05.893604 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 22766
I20260928 16:53:05.894451 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.93.20:45181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:36935
--local_ip_for_outbound_sockets=127.9.93.3
--tserver_master_addrs=127.9.93.62:42761
--webserver_port=37593
--webserver_interface=127.9.93.3
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:38517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.93.20:45181/kms with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:53:05 dist-test-slave-wzcb krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614385, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:53:05.984810 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:05.912727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:47319 (local address 127.9.93.62:42761)
0928 16:53:05.941698 (+ 28971us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:05.941708 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:05.947411 (+ 5703us) server_negotiation.cc:407] Connection header received
0928 16:53:05.951992 (+ 4581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:05.952000 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:05.952124 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:05.952235 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:05.964078 (+ 11843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:05.965334 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:05.968310 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:05.969039 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:05.969159 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:05.975081 (+ 5922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:05.975091 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:05.975096 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:05.975155 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:05.979412 (+ 4257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:05.980866 (+ 1454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:05.980874 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:05.980878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:05.981083 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:05.981963 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:05.981969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:05.981973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:05.982427 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:05.982609 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:05.983943 (+ 1334us) server_negotiation.cc:299] Negotiation successful
0928 16:53:05.984279 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28755,"thread_start_us":28993,"threads_started":1}
I20260928 16:53:05.988225 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:05.911685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42761 (local address 127.9.93.2:47319)
0928 16:53:05.942684 (+ 30999us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:05.942772 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:05.951525 (+ 8753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:05.961512 (+ 9987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:05.961536 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:05.962419 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:53:05.963732 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:05.963774 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:05.965662 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:05.965669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:05.968058 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:05.968082 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:05.969641 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:05.969650 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:05.970007 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:05.971050 (+ 1043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:53:05.971099 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:53:05.974669 (+ 3570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:53:05.979750 (+ 5081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:05.979763 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:05.979787 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:05.980649 (+ 862us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:05.981282 (+ 633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:05.981287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:05.981292 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:05.981675 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:05.982729 (+ 1054us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:53:05.982739 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:53:05.983351 (+ 612us) client_negotiation.cc:769] Sending connection context
0928 16:53:05.985501 (+ 2150us) client_negotiation.cc:240] Negotiation successful
0928 16:53:05.986318 (+ 817us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30485,"thread_start_us":35538,"threads_started":1}
I20260928 16:53:05.993563 22908 heartbeater.cc:347] Connected to a master server at 127.9.93.62:42761
I20260928 16:53:05.993965 22908 heartbeater.cc:464] Registering TS with master...
I20260928 16:53:05.995087 22908 heartbeater.cc:511] Master 127.9.93.62:42761 requested a full tablet report, sending...
I20260928 16:53:05.998016 22564 ts_manager.cc:194] Registered new tserver with Master: aa38467beec143adacf51cfc98e694fc (127.9.93.2:45165)
I20260928 16:53:06.001029 22564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.2@KRBTEST.COM'} at 127.9.93.2:47319
I20260928 16:53:06.074878 22762 heartbeater.cc:464] Registering TS with master...
I20260928 16:53:06.075435 22762 heartbeater.cc:511] Master 127.9.93.62:42761 requested a full tablet report, sending...
I20260928 16:53:06.076907 22564 ts_manager.cc:194] Registered new tserver with Master: d63822f0058c43f6b61ea89c631373c6 (127.9.93.1:34831)
I20260928 16:53:06.078194 22564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.1@KRBTEST.COM'} at 127.9.93.1:35107
Sep 28 16:53:06 dist-test-slave-wzcb krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614385, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for HTTP/127.9.93.20@KRBTEST.COM
W20260928 16:53:06.531476 22909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:06.531865 22909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:06.532042 22909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:06.540254 22909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:06.540434 22909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:06.540618 22909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:53:06.555625 22909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:38517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.93.20:45181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:36935
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=37593
--tserver_master_addrs=127.9.93.62:42761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:06.557588 22909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:06.559604 22909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:06.580113 22918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:06.580755 22917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:06.589290 22909 server_base.cc:1034] running on GCE node
W20260928 16:53:06.589718 22920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:06.590677 22909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:06.592448 22909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:06.593884 22909 hybrid_clock.cc:648] HybridClock initialized: now 1790614386593781 us; error 80 us; skew 500 ppm
I20260928 16:53:06.594359 22909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:06 dist-test-slave-wzcb krb5kdc[21552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614386, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:06.599393 22909 init.cc:377] Logged in from keytab as kudu/127.9.93.3@KRBTEST.COM (short username kudu)
I20260928 16:53:06.601994 22909 webserver.cc:460] Webserver started at http://127.9.93.3:37593/ using document root <none> and password file <none>
I20260928 16:53:06.603135 22909 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:06.603356 22909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 16:53:06 dist-test-slave-wzcb krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614386, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for HTTP/127.9.93.20@KRBTEST.COM
I20260928 16:53:06.639786 22909 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:53:06.649174 22928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:06.651929 22909 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.006s sys 0.003s
I20260928 16:53:06.652295 22909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "83c97d77def145cea0d9d317211e7b88"
format_stamp: "Formatted at 2026-09-28 16:53:02 on dist-test-slave-wzcb"
server_key: "374649a12dec53a97303fefa59434876"
server_key_iv: "f93f0d6ac4ccb9b24f52abc05d857522"
server_key_version: "kuduclusterkey@0"
I20260928 16:53:06.654251 22909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:06.675973 22909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:06.682224 22909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:06.682669 22909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:06.687163 22909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:53:06.689549 22909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:53:06.689785 22909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 16:53:06.689955 22909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:53:06.690176 22909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:53:06 dist-test-slave-wzcb krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614385, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for HTTP/127.9.93.20@KRBTEST.COM
I20260928 16:53:07.008059 22908 heartbeater.cc:503] Master 127.9.93.62:42761 was elected leader, sending a full tablet report...
I20260928 16:53:07.022203 22909 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:36935
I20260928 16:53:07.022311 23041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:36935 every 8 connection(s)
I20260928 16:53:07.024679 22909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:53:07.027264 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 22909
I20260928 16:53:07.070425 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:07.032947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33922 (local address 127.9.93.62:42761)
0928 16:53:07.041187 (+ 8240us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:07.041195 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:07.041219 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:53:07.041317 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:07.041324 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:07.041422 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:07.041557 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:07.044034 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.045718 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.052857 (+ 7139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.053714 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.053890 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:07.061462 (+ 7572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:07.061485 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:07.061492 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:07.061666 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:07.065819 (+ 4153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:07.066966 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:07.066973 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:07.066977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:07.067225 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:07.067889 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:07.067895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:07.067898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:07.068337 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:07.068636 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:07.069721 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0928 16:53:07.070036 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8094,"thread_start_us":7953,"threads_started":1}
Sep 28 16:53:07 dist-test-slave-wzcb krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614386, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:53:07.082510 22762 heartbeater.cc:503] Master 127.9.93.62:42761 was elected leader, sending a full tablet report...
I20260928 16:53:07.083125 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:07.045112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:33291 (local address 127.9.93.62:42761)
0928 16:53:07.057389 (+ 12277us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:07.057396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:07.057417 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:53:07.057506 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:07.057511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:07.057621 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:07.057751 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:07.060608 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.063774 (+ 3166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.066790 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.067555 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.067687 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:07.074192 (+ 6505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:07.074202 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:07.074209 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:07.074270 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:07.077677 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:07.079083 (+ 1406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:07.079090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:07.079092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:07.079460 (+ 368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:07.080290 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:07.080295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:07.080298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:07.080691 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:07.080832 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:07.082180 (+ 1348us) server_negotiation.cc:299] Negotiation successful
0928 16:53:07.082510 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12106,"thread_start_us":12287,"threads_started":1}
I20260928 16:53:07.084676 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:07.044164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42761 (local address 127.9.93.3:33291)
0928 16:53:07.053857 (+ 9693us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:07.053954 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:07.056912 (+ 2958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:07.058307 (+ 1395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:07.058322 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:07.059070 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:53:07.060228 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:07.060267 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.064145 (+ 3878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.064153 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:07.066533 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:07.066561 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.068153 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.068165 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:07.068577 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:07.069484 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:53:07.069525 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:53:07.073669 (+ 4144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:53:07.077959 (+ 4290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:07.077974 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:07.077999 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:07.078843 (+ 844us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:07.079661 (+ 818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:07.079669 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:07.079673 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:07.080096 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:07.080887 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:53:07.080899 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:53:07.081523 (+ 624us) client_negotiation.cc:769] Sending connection context
0928 16:53:07.081974 (+ 451us) client_negotiation.cc:240] Negotiation successful
0928 16:53:07.082769 (+ 795us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9181,"thread_start_us":11297,"threads_started":1}
I20260928 16:53:07.086529 23042 heartbeater.cc:347] Connected to a master server at 127.9.93.62:42761
I20260928 16:53:07.086982 23042 heartbeater.cc:464] Registering TS with master...
I20260928 16:53:07.088289 23042 heartbeater.cc:511] Master 127.9.93.62:42761 requested a full tablet report, sending...
I20260928 16:53:07.090444 22564 ts_manager.cc:194] Registered new tserver with Master: 83c97d77def145cea0d9d317211e7b88 (127.9.93.3:36935)
I20260928 16:53:07.091416 22564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.3@KRBTEST.COM'} at 127.9.93.3:33291
I20260928 16:53:07.102072 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:53:07.121848 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:07.106109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33932 (local address 127.9.93.62:42761)
0928 16:53:07.106320 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:07.106330 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:07.106792 (+ 462us) server_negotiation.cc:407] Connection header received
0928 16:53:07.107005 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:07.107012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:07.107125 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:07.107224 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:07.109096 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.109891 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.111450 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.111951 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.112082 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:07.115132 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:07.115138 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:07.115142 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:07.115192 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:07.118450 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:07.119492 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:07.119498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:07.119501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:07.119687 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:07.120187 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:07.120192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:07.120194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:07.120546 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:07.120819 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:07.121215 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 16:53:07.121500 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:53:07.128358 22564 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:53:07.137360 22564 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:53:07.181535 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:07.155530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:34831 (local address 127.0.0.1:40756)
0928 16:53:07.168531 (+ 13001us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:07.168593 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:07.168893 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:07.171290 (+ 2397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:07.171320 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:07.171384 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:53:07.171936 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:07.171955 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.174484 (+ 2529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.174491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:07.177442 (+ 2951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:07.177470 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.179937 (+ 2467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.179944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:07.180240 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:07.180333 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 16:53:07.180479 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 16:53:07.181138 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11551,"thread_start_us":14916,"threads_started":1}
I20260928 16:53:07.184113 23065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:07.155910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40756 (local address 127.9.93.1:34831)
0928 16:53:07.170349 (+ 14439us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:07.170361 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:07.170404 (+ 43us) server_negotiation.cc:407] Connection header received
0928 16:53:07.170552 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:07.170563 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:07.170980 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:07.172172 (+ 1192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:53:07.172523 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.174264 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.177775 (+ 3511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.179634 (+ 1859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.182937 (+ 3303us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:07.183271 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:07.183424 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 16:53:07.183680 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14218,"thread_start_us":14020,"threads_started":1}
I20260928 16:53:07.186393 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:07.156525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43246 (local address 127.9.93.2:45165)
0928 16:53:07.169735 (+ 13210us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:07.169747 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:07.170981 (+ 1234us) server_negotiation.cc:407] Connection header received
0928 16:53:07.171437 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:07.171447 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:07.171742 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:07.171957 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:53:07.173071 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.174733 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.181135 (+ 6402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.182981 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.183142 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:07.183459 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:07.185669 (+ 2210us) server_negotiation.cc:299] Negotiation successful
0928 16:53:07.185953 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12971,"thread_start_us":12733,"threads_started":1}
I20260928 16:53:07.187474 23064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:07.156143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:45165 (local address 127.0.0.1:43246)
0928 16:53:07.170798 (+ 14655us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:07.170823 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:07.171130 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:07.172307 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:07.172312 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:07.172455 (+ 143us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:53:07.172818 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:07.172830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.174947 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.174953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:07.180778 (+ 5825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:07.180796 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.183760 (+ 2964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.183768 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:07.184020 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:07.184064 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 16:53:07.186776 (+ 2712us) client_negotiation.cc:240] Negotiation successful
0928 16:53:07.187002 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14502,"thread_start_us":11215,"threads_started":1}
I20260928 16:53:07.189558 22697 tablet_service.cc:1467] Processing CreateTablet for tablet 1112307f23c54bd7a6bc60629852cd72 (DEFAULT_TABLE table=test-table [id=e78d1982e0604bb08c63244082ae9f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:53:07.191004 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:07.175556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:36935 (local address 127.0.0.1:51860)
0928 16:53:07.177886 (+ 2330us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:07.177908 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:07.178081 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:07.179180 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:07.179186 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:07.179219 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:53:07.179799 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:07.179813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.183360 (+ 3547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.183366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:07.185351 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:07.185367 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.189849 (+ 4482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.189865 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:07.190111 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:07.190153 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 16:53:07.190245 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 16:53:07.190420 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2111,"mutex_wait_us":1,"thread_start_us":1890,"threads_started":1}
I20260928 16:53:07.191957 22697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1112307f23c54bd7a6bc60629852cd72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:53:07.194852 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:07.172229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51860 (local address 127.9.93.3:36935)
0928 16:53:07.177353 (+ 5124us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:07.177364 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:07.178233 (+ 869us) server_negotiation.cc:407] Connection header received
0928 16:53:07.178380 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:07.178389 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:07.178898 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:07.180076 (+ 1178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:53:07.180461 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.183139 (+ 2678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.185585 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.189420 (+ 3835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.191798 (+ 2378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:07.194089 (+ 2291us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:07.194200 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 16:53:07.194425 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4911,"thread_start_us":4757,"threads_started":1}
I20260928 16:53:07.198659 22841 tablet_service.cc:1467] Processing CreateTablet for tablet 1112307f23c54bd7a6bc60629852cd72 (DEFAULT_TABLE table=test-table [id=e78d1982e0604bb08c63244082ae9f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:53:07.201267 22841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1112307f23c54bd7a6bc60629852cd72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:53:07.201267 22977 tablet_service.cc:1467] Processing CreateTablet for tablet 1112307f23c54bd7a6bc60629852cd72 (DEFAULT_TABLE table=test-table [id=e78d1982e0604bb08c63244082ae9f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:53:07.204891 22977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1112307f23c54bd7a6bc60629852cd72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:53:07.228274 23070 tablet_bootstrap.cc:492] T 1112307f23c54bd7a6bc60629852cd72 P d63822f0058c43f6b61ea89c631373c6: Bootstrap starting.
I20260928 16:53:07.233809 23070 tablet_bootstrap.cc:654] T 1112307f23c54bd7a6bc60629852cd72 P d63822f0058c43f6b61ea89c631373c6: Neither blocks nor log segments found. Creating new log.
I20260928 16:53:07.235185 23070 log.cc:824] T 1112307f23c54bd7a6bc60629852cd72 P d63822f0058c43f6b61ea89c631373c6: Log is configured to *not* fsync() on all Append() calls
I20260928 16:53:07.238034 23072 tablet_bootstrap.cc:492] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88: Bootstrap starting.
I20260928 16:53:07.241779 23071 tablet_bootstrap.cc:492] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc: Bootstrap starting.
I20260928 16:53:07.246407 23071 tablet_bootstrap.cc:654] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc: Neither blocks nor log segments found. Creating new log.
I20260928 16:53:07.246411 23072 tablet_bootstrap.cc:654] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88: Neither blocks nor log segments found. Creating new log.
I20260928 16:53:07.246604 23070 tablet_bootstrap.cc:492] T 1112307f23c54bd7a6bc60629852cd72 P d63822f0058c43f6b61ea89c631373c6: No bootstrap required, opened a new log
I20260928 16:53:07.246979 23070 ts_tablet_manager.cc:1397] T 1112307f23c54bd7a6bc60629852cd72 P d63822f0058c43f6b61ea89c631373c6: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.005s
I20260928 16:53:07.248099 23071 log.cc:824] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc: Log is configured to *not* fsync() on all Append() calls
I20260928 16:53:07.248135 23072 log.cc:824] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88: Log is configured to *not* fsync() on all Append() calls
I20260928 16:53:07.259114 23072 tablet_bootstrap.cc:492] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88: No bootstrap required, opened a new log
I20260928 16:53:07.259570 23071 tablet_bootstrap.cc:492] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc: No bootstrap required, opened a new log
I20260928 16:53:07.259222 23070 raft_consensus.cc:348] T 1112307f23c54bd7a6bc60629852cd72 P d63822f0058c43f6b61ea89c631373c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63822f0058c43f6b61ea89c631373c6" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 34831 } } peers { permanent_uuid: "83c97d77def145cea0d9d317211e7b88" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36935 } } peers { permanent_uuid: "aa38467beec143adacf51cfc98e694fc" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 45165 } }
I20260928 16:53:07.259709 23072 ts_tablet_manager.cc:1397] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.004s
I20260928 16:53:07.259877 23070 raft_consensus.cc:374] T 1112307f23c54bd7a6bc60629852cd72 P d63822f0058c43f6b61ea89c631373c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:53:07.259903 23071 ts_tablet_manager.cc:1397] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.009s
I20260928 16:53:07.260001 23070 raft_consensus.cc:729] T 1112307f23c54bd7a6bc60629852cd72 P d63822f0058c43f6b61ea89c631373c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d63822f0058c43f6b61ea89c631373c6, State: Initialized, Role: FOLLOWER
I20260928 16:53:07.261018 23070 consensus_queue.cc:260] T 1112307f23c54bd7a6bc60629852cd72 P d63822f0058c43f6b61ea89c631373c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63822f0058c43f6b61ea89c631373c6" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 34831 } } peers { permanent_uuid: "83c97d77def145cea0d9d317211e7b88" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36935 } } peers { permanent_uuid: "aa38467beec143adacf51cfc98e694fc" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 45165 } }
I20260928 16:53:07.273073 23071 raft_consensus.cc:348] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63822f0058c43f6b61ea89c631373c6" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 34831 } } peers { permanent_uuid: "83c97d77def145cea0d9d317211e7b88" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36935 } } peers { permanent_uuid: "aa38467beec143adacf51cfc98e694fc" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 45165 } }
I20260928 16:53:07.273782 23071 raft_consensus.cc:374] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:53:07.273914 23071 raft_consensus.cc:729] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa38467beec143adacf51cfc98e694fc, State: Initialized, Role: FOLLOWER
I20260928 16:53:07.273499 23072 raft_consensus.cc:348] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63822f0058c43f6b61ea89c631373c6" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 34831 } } peers { permanent_uuid: "83c97d77def145cea0d9d317211e7b88" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36935 } } peers { permanent_uuid: "aa38467beec143adacf51cfc98e694fc" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 45165 } }
I20260928 16:53:07.274262 23072 raft_consensus.cc:374] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:53:07.274408 23072 raft_consensus.cc:729] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c97d77def145cea0d9d317211e7b88, State: Initialized, Role: FOLLOWER
I20260928 16:53:07.275352 23072 consensus_queue.cc:260] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63822f0058c43f6b61ea89c631373c6" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 34831 } } peers { permanent_uuid: "83c97d77def145cea0d9d317211e7b88" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36935 } } peers { permanent_uuid: "aa38467beec143adacf51cfc98e694fc" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 45165 } }
I20260928 16:53:07.276266 23070 ts_tablet_manager.cc:1428] T 1112307f23c54bd7a6bc60629852cd72 P d63822f0058c43f6b61ea89c631373c6: Time spent starting tablet: real 0.029s user 0.018s sys 0.013s
I20260928 16:53:07.277176 23071 consensus_queue.cc:260] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63822f0058c43f6b61ea89c631373c6" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 34831 } } peers { permanent_uuid: "83c97d77def145cea0d9d317211e7b88" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36935 } } peers { permanent_uuid: "aa38467beec143adacf51cfc98e694fc" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 45165 } }
I20260928 16:53:07.283215 23072 ts_tablet_manager.cc:1428] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88: Time spent starting tablet: real 0.023s user 0.010s sys 0.007s
I20260928 16:53:07.283816 23042 heartbeater.cc:503] Master 127.9.93.62:42761 was elected leader, sending a full tablet report...
I20260928 16:53:07.286546 23071 ts_tablet_manager.cc:1428] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc: Time spent starting tablet: real 0.027s user 0.013s sys 0.010s
W20260928 16:53:07.291877 23043 tablet.cc:2369] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:53:07.408583 23078 raft_consensus.cc:482] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:53:07.408964 23078 raft_consensus.cc:504] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63822f0058c43f6b61ea89c631373c6" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 34831 } } peers { permanent_uuid: "83c97d77def145cea0d9d317211e7b88" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36935 } } peers { permanent_uuid: "aa38467beec143adacf51cfc98e694fc" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 45165 } }
I20260928 16:53:07.411087 23078 leader_election.cc:288] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d63822f0058c43f6b61ea89c631373c6 (127.9.93.1:34831), 83c97d77def145cea0d9d317211e7b88 (127.9.93.3:36935)
I20260928 16:53:07.417757 23077 raft_consensus.cc:482] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:53:07.418146 23077 raft_consensus.cc:504] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63822f0058c43f6b61ea89c631373c6" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 34831 } } peers { permanent_uuid: "83c97d77def145cea0d9d317211e7b88" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36935 } } peers { permanent_uuid: "aa38467beec143adacf51cfc98e694fc" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 45165 } }
I20260928 16:53:07.420440 23065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:07.411606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:59803 (local address 127.9.93.1:34831)
0928 16:53:07.411877 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:07.411885 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:07.412362 (+ 477us) server_negotiation.cc:407] Connection header received
0928 16:53:07.412670 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:07.412678 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:07.412785 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:07.412858 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:53:07.413772 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.414728 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.417522 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.418866 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.418974 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:07.419100 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:07.419949 (+ 849us) server_negotiation.cc:299] Negotiation successful
0928 16:53:07.420136 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 16:53:07.421329 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:07.412844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:39781 (local address 127.9.93.3:36935)
0928 16:53:07.413070 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:07.413076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:07.413850 (+ 774us) server_negotiation.cc:407] Connection header received
0928 16:53:07.414022 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:07.414027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:07.414123 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:07.414224 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:53:07.414962 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.416158 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.418691 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.420024 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.420137 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:07.420247 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:07.420881 (+ 634us) server_negotiation.cc:299] Negotiation successful
0928 16:53:07.421057 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 16:53:07.421823 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:07.411446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:34831 (local address 127.9.93.2:59803)
0928 16:53:07.412237 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:07.412257 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:07.412430 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:07.413115 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:07.413121 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:07.413160 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:53:07.413617 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:07.413629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.414871 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.414877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:07.417272 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:07.417293 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.419497 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.419504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:07.419715 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:07.419756 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 16:53:07.420976 (+ 1220us) client_negotiation.cc:240] Negotiation successful
0928 16:53:07.421154 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":1,"thread_start_us":421,"threads_started":1}
I20260928 16:53:07.422103 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:07.412450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:36935 (local address 127.9.93.2:39781)
0928 16:53:07.413704 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:07.413725 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:07.413881 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:07.414271 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:07.414276 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:07.414306 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:53:07.414766 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:07.414780 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.416361 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.416367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:07.418467 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:07.418489 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.420465 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.420471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:07.420663 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:07.420698 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:53:07.421542 (+ 844us) client_negotiation.cc:240] Negotiation successful
0928 16:53:07.421710 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1096,"thread_start_us":834,"threads_started":1}
I20260928 16:53:07.423271 22717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1112307f23c54bd7a6bc60629852cd72" candidate_uuid: "aa38467beec143adacf51cfc98e694fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63822f0058c43f6b61ea89c631373c6" is_pre_election: true
I20260928 16:53:07.423281 22997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1112307f23c54bd7a6bc60629852cd72" candidate_uuid: "aa38467beec143adacf51cfc98e694fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c97d77def145cea0d9d317211e7b88" is_pre_election: true
I20260928 16:53:07.423956 22997 raft_consensus.cc:2454] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa38467beec143adacf51cfc98e694fc in term 0.
I20260928 16:53:07.424014 22717 raft_consensus.cc:2454] T 1112307f23c54bd7a6bc60629852cd72 P d63822f0058c43f6b61ea89c631373c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa38467beec143adacf51cfc98e694fc in term 0.
I20260928 16:53:07.424932 22797 leader_election.cc:302] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa38467beec143adacf51cfc98e694fc, d63822f0058c43f6b61ea89c631373c6; no voters:
I20260928 16:53:07.425980 23078 raft_consensus.cc:2781] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:53:07.426179 23078 raft_consensus.cc:482] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:53:07.426287 23078 raft_consensus.cc:3037] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:53:07.430569 23078 raft_consensus.cc:504] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63822f0058c43f6b61ea89c631373c6" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 34831 } } peers { permanent_uuid: "83c97d77def145cea0d9d317211e7b88" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36935 } } peers { permanent_uuid: "aa38467beec143adacf51cfc98e694fc" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 45165 } }
I20260928 16:53:07.431226 23077 leader_election.cc:288] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d63822f0058c43f6b61ea89c631373c6 (127.9.93.1:34831), aa38467beec143adacf51cfc98e694fc (127.9.93.2:45165)
I20260928 16:53:07.432642 22717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1112307f23c54bd7a6bc60629852cd72" candidate_uuid: "aa38467beec143adacf51cfc98e694fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63822f0058c43f6b61ea89c631373c6"
I20260928 16:53:07.433167 22717 raft_consensus.cc:3037] T 1112307f23c54bd7a6bc60629852cd72 P d63822f0058c43f6b61ea89c631373c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:53:07.433450 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:07.423751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:34831 (local address 127.9.93.3:34691)
0928 16:53:07.424209 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:07.424229 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:07.424367 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:07.425452 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:07.425458 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:07.425499 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:53:07.425714 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:07.425726 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.427850 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.427856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:07.430043 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:07.430070 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.432511 (+ 2441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.432519 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:07.432753 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:07.432796 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 16:53:07.432918 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 16:53:07.433091 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321}
I20260928 16:53:07.437166 23078 leader_election.cc:288] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc [CANDIDATE]: Term 1 election: Requested vote from peers d63822f0058c43f6b61ea89c631373c6 (127.9.93.1:34831), 83c97d77def145cea0d9d317211e7b88 (127.9.93.3:36935)
I20260928 16:53:07.437907 22997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1112307f23c54bd7a6bc60629852cd72" candidate_uuid: "aa38467beec143adacf51cfc98e694fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c97d77def145cea0d9d317211e7b88"
I20260928 16:53:07.438303 23065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:07.424753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:34691 (local address 127.9.93.1:34831)
0928 16:53:07.424951 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:07.424957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:07.424977 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:53:07.425047 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:07.425052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:07.425145 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:07.425270 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:53:07.426416 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.427637 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.430342 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.432287 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.437562 (+ 5275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:07.437697 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:07.437766 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 16:53:07.437947 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:53:07.438256 22997 raft_consensus.cc:3037] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:53:07.439002 22717 raft_consensus.cc:2454] T 1112307f23c54bd7a6bc60629852cd72 P d63822f0058c43f6b61ea89c631373c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa38467beec143adacf51cfc98e694fc in term 1.
I20260928 16:53:07.439142 22716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1112307f23c54bd7a6bc60629852cd72" candidate_uuid: "83c97d77def145cea0d9d317211e7b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63822f0058c43f6b61ea89c631373c6" is_pre_election: true
I20260928 16:53:07.439668 22716 raft_consensus.cc:2379] T 1112307f23c54bd7a6bc60629852cd72 P d63822f0058c43f6b61ea89c631373c6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83c97d77def145cea0d9d317211e7b88 in current term 1: Already voted for candidate aa38467beec143adacf51cfc98e694fc in this term.
I20260928 16:53:07.439826 22797 leader_election.cc:302] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa38467beec143adacf51cfc98e694fc, d63822f0058c43f6b61ea89c631373c6; no voters:
I20260928 16:53:07.440310 23078 raft_consensus.cc:2781] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:53:07.442485 22997 raft_consensus.cc:2454] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa38467beec143adacf51cfc98e694fc in term 1.
I20260928 16:53:07.443738 23078 raft_consensus.cc:686] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc [term 1 LEADER]: Becoming Leader. State: Replica: aa38467beec143adacf51cfc98e694fc, State: Running, Role: LEADER
I20260928 16:53:07.444432 23078 consensus_queue.cc:237] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63822f0058c43f6b61ea89c631373c6" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 34831 } } peers { permanent_uuid: "83c97d77def145cea0d9d317211e7b88" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36935 } } peers { permanent_uuid: "aa38467beec143adacf51cfc98e694fc" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 45165 } }
I20260928 16:53:07.444756 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:07.426062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:42707 (local address 127.9.93.2:45165)
0928 16:53:07.426303 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:07.426308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:07.434035 (+ 7727us) server_negotiation.cc:407] Connection header received
0928 16:53:07.434411 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:07.434417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:07.434539 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:07.434611 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:53:07.435265 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.436461 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.441261 (+ 4800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.442774 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.442913 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:07.443037 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:07.443791 (+ 754us) server_negotiation.cc:299] Negotiation successful
0928 16:53:07.444347 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 16:53:07.445086 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:07.425205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:45165 (local address 127.9.93.3:42707)
0928 16:53:07.433925 (+ 8720us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:07.433946 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:07.434184 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:07.434842 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:07.434847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:07.434878 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:53:07.435087 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:07.435099 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.436679 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.436685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:07.441027 (+ 4342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:07.441047 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.443214 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.443221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:07.443567 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:07.443607 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 16:53:07.444476 (+ 869us) client_negotiation.cc:240] Negotiation successful
0928 16:53:07.444655 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8577,"thread_start_us":11302,"threads_started":1}
I20260928 16:53:07.446216 22861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1112307f23c54bd7a6bc60629852cd72" candidate_uuid: "83c97d77def145cea0d9d317211e7b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa38467beec143adacf51cfc98e694fc" is_pre_election: true
I20260928 16:53:07.447657 22930 leader_election.cc:302] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83c97d77def145cea0d9d317211e7b88; no voters: aa38467beec143adacf51cfc98e694fc, d63822f0058c43f6b61ea89c631373c6
I20260928 16:53:07.449417 23077 raft_consensus.cc:2726] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 16:53:07.450832 22564 catalog_manager.cc:5495] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc reported cstate change: term changed from 0 to 1, leader changed from <none> to aa38467beec143adacf51cfc98e694fc (127.9.93.2). New cstate: current_term: 1 leader_uuid: "aa38467beec143adacf51cfc98e694fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63822f0058c43f6b61ea89c631373c6" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 34831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c97d77def145cea0d9d317211e7b88" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa38467beec143adacf51cfc98e694fc" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 45165 } health_report { overall_health: HEALTHY } } }
W20260928 16:53:07.463114 22910 tablet.cc:2369] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:53:07.494678 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:07.488408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43250 (local address 127.9.93.2:45165)
0928 16:53:07.488627 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:07.488634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:07.488656 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:53:07.488790 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:07.488794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:07.488907 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:07.489062 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:53:07.489535 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.490378 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.492034 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.492564 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.492694 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:07.493336 (+ 642us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:53:07.493971 (+ 635us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:53:07.494127 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:07.494232 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 16:53:07.494347 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 16:53:07.496277 22763 tablet.cc:2369] T 1112307f23c54bd7a6bc60629852cd72 P d63822f0058c43f6b61ea89c631373c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:53:07.505872 22997 raft_consensus.cc:1261] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88 [term 1 FOLLOWER]: Refusing update from remote peer aa38467beec143adacf51cfc98e694fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:53:07.505923 22716 raft_consensus.cc:1261] T 1112307f23c54bd7a6bc60629852cd72 P d63822f0058c43f6b61ea89c631373c6 [term 1 FOLLOWER]: Refusing update from remote peer aa38467beec143adacf51cfc98e694fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:53:07.507093 23082 consensus_queue.cc:1035] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d63822f0058c43f6b61ea89c631373c6" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 34831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:53:07.507620 23078 consensus_queue.cc:1035] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c97d77def145cea0d9d317211e7b88" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:53:07.554577 22564 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:53:07.554936 22564 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:53:07.560689 22564 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c4a7d424812445f87c874e9c2732c6a: Sending DeleteTablet for 3 replicas of tablet 1112307f23c54bd7a6bc60629852cd72
I20260928 16:53:07.562479 22977 tablet_service.cc:1514] Processing DeleteTablet for tablet 1112307f23c54bd7a6bc60629852cd72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:53:07 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:51860
I20260928 16:53:07.563068 22697 tablet_service.cc:1514] Processing DeleteTablet for tablet 1112307f23c54bd7a6bc60629852cd72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:53:07 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:40756
I20260928 16:53:07.567759 22841 tablet_service.cc:1514] Processing DeleteTablet for tablet 1112307f23c54bd7a6bc60629852cd72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:53:07 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:43246
I20260928 16:53:07.574918 23095 tablet_replica.cc:330] T 1112307f23c54bd7a6bc60629852cd72 P d63822f0058c43f6b61ea89c631373c6: stopping tablet replica
I20260928 16:53:07.575583 23095 raft_consensus.cc:2229] T 1112307f23c54bd7a6bc60629852cd72 P d63822f0058c43f6b61ea89c631373c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:53:07.576275 23095 raft_consensus.cc:2258] T 1112307f23c54bd7a6bc60629852cd72 P d63822f0058c43f6b61ea89c631373c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:53:07.575258 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 22601
I20260928 16:53:07.595602 23095 ts_tablet_manager.cc:1905] T 1112307f23c54bd7a6bc60629852cd72 P d63822f0058c43f6b61ea89c631373c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:53:07.605505 23096 tablet_replica.cc:330] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc: stopping tablet replica
I20260928 16:53:07.611994 23096 raft_consensus.cc:2229] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:53:07.613212 23096 raft_consensus.cc:2258] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:53:07.616775 23094 tablet_replica.cc:330] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88: stopping tablet replica
I20260928 16:53:07.618453 23096 ts_tablet_manager.cc:1905] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:53:07.621577 23094 raft_consensus.cc:2229] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:53:07.625432 23095 ts_tablet_manager.cc:1918] T 1112307f23c54bd7a6bc60629852cd72 P d63822f0058c43f6b61ea89c631373c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:53:07.626572 23094 raft_consensus.cc:2258] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:53:07.627748 23096 ts_tablet_manager.cc:1918] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:53:07.628031 23096 log.cc:1196] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/1112307f23c54bd7a6bc60629852cd72
I20260928 16:53:07.628697 23096 ts_tablet_manager.cc:1939] T 1112307f23c54bd7a6bc60629852cd72 P aa38467beec143adacf51cfc98e694fc: Deleting consensus metadata
I20260928 16:53:07.631215 22549 catalog_manager.cc:4841] TS aa38467beec143adacf51cfc98e694fc (127.9.93.2:45165): tablet 1112307f23c54bd7a6bc60629852cd72 (table test-table [id=e78d1982e0604bb08c63244082ae9f87]) successfully deleted
I20260928 16:53:07.631925 23095 log.cc:1196] T 1112307f23c54bd7a6bc60629852cd72 P d63822f0058c43f6b61ea89c631373c6: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-0/wal/wals/1112307f23c54bd7a6bc60629852cd72
I20260928 16:53:07.632596 23095 ts_tablet_manager.cc:1939] T 1112307f23c54bd7a6bc60629852cd72 P d63822f0058c43f6b61ea89c631373c6: Deleting consensus metadata
I20260928 16:53:07.634871 22551 catalog_manager.cc:4841] TS d63822f0058c43f6b61ea89c631373c6 (127.9.93.1:34831): tablet 1112307f23c54bd7a6bc60629852cd72 (table test-table [id=e78d1982e0604bb08c63244082ae9f87]) successfully deleted
I20260928 16:53:07.636193 23094 ts_tablet_manager.cc:1905] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:53:07.644163 23094 ts_tablet_manager.cc:1918] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:53:07.644464 23094 log.cc:1196] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/1112307f23c54bd7a6bc60629852cd72
I20260928 16:53:07.645097 23094 ts_tablet_manager.cc:1939] T 1112307f23c54bd7a6bc60629852cd72 P 83c97d77def145cea0d9d317211e7b88: Deleting consensus metadata
I20260928 16:53:07.645612 23065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:07.577362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40762 (local address 127.9.93.1:34831)
0928 16:53:07.577856 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:07.577864 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:07.604262 (+ 26398us) server_negotiation.cc:407] Connection header received
0928 16:53:07.605889 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:07.605898 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:07.606022 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:07.606131 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:07.613652 (+ 7521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.617400 (+ 3748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.627003 (+ 9603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.628021 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.628189 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:07.636279 (+ 8090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:07.636301 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:07.636323 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:07.636384 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:07.641008 (+ 4624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:07.642241 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:07.642255 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:07.642259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:07.642470 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:07.643040 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:07.643046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:07.643049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:07.643602 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:07.643955 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:07.644802 (+ 847us) server_negotiation.cc:299] Negotiation successful
0928 16:53:07.645146 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327}
I20260928 16:53:07.647519 22551 catalog_manager.cc:4841] TS 83c97d77def145cea0d9d317211e7b88 (127.9.93.3:36935): tablet 1112307f23c54bd7a6bc60629852cd72 (table test-table [id=e78d1982e0604bb08c63244082ae9f87]) successfully deleted
I20260928 16:53:07.647662 22757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:07.750149 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 22601
I20260928 16:53:07.768992 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 22766
I20260928 16:53:07.787571 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:07.769984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43260 (local address 127.9.93.2:45165)
0928 16:53:07.770626 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:07.770631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:07.770651 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:53:07.770719 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:07.770724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:07.770831 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:07.770948 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:07.772655 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.773550 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.775490 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.776061 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.776224 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:07.779287 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:07.779363 (+ 76us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:07.779390 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:07.779448 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:07.783244 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:07.784314 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:07.784326 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:07.784334 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:07.784554 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:07.785223 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:07.785229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:07.785231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:07.785858 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:07.786300 (+ 442us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:07.786689 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 16:53:07.787043 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500}
I20260928 16:53:07.788604 22903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:07.919772 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 22766
I20260928 16:53:07.938969 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 22909
I20260928 16:53:07.956954 23100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:07.939902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51876 (local address 127.9.93.3:36935)
0928 16:53:07.940444 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:07.940449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:07.940470 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:53:07.940548 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:07.940552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:07.940622 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:07.940731 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:07.942648 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.943529 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.945321 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:07.946066 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:07.946204 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:07.949927 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:07.949949 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:07.949968 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:07.950042 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:07.953187 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:07.954333 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:07.954344 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:07.954347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:07.954583 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:07.955076 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:07.955079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:07.955081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:07.955672 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:07.956038 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:07.956378 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 16:53:07.956611 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":289,"threads_started":1}
I20260928 16:53:07.957695 23037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:08.057129 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 22909
I20260928 16:53:08.072906 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 22531
I20260928 16:53:08.074339 22594 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:08.206132 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 22531
I20260928 16:53:08.228387 9588 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 16:53:08.575242 9588 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 16:53:08.575531 9588 mini_ranger.cc:66] Stopping Ranger...
I20260928 16:53:08.922732 9588 mini_ranger.cc:68] Stopped Ranger
2026-09-28 16:53:08.923 UTC [21582] LOG: received smart shutdown request
2026-09-28 16:53:08.927 UTC [21582] LOG: background worker "logical replication launcher" (PID 21591) exited with exit code 1
2026-09-28 16:53:08.928 UTC [21586] LOG: shutting down
2026-09-28 16:53:09.142 UTC [21582] LOG: database system is shut down
2026-09-28T16:53:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (58736 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:53:09 dist-test-slave-wzcb krb5kdc[23111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:53:09 dist-test-slave-wzcb krb5kdc[23111](info): set up 2 sockets
Sep 28 16:53:09 dist-test-slave-wzcb krb5kdc[23111](info): commencing operation
krb5kdc: starting...
W20260928 16:53:10.322219 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:53:10 dist-test-slave-wzcb krb5kdc[23111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:53:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:53:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:53:10.591663 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:45901
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:35745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:45901,127.9.93.61:44891,127.9.93.60:37545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:53:10.979928 23124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:10.980275 23124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:10.980427 23124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:10.988387 23124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:53:10.988521 23124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:10.988575 23124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:10.988660 23124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:53:10.988721 23124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:53:10.999963 23124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:35745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:45901,127.9.93.61:44891,127.9.93.60:37545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:45901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:11.001710 23124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:11.003394 23124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:11.017019 23132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:11.017144 23129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:11.017078 23130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:11.018594 23124 server_base.cc:1034] running on GCE node
I20260928 16:53:11.019697 23124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:11.021560 23124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:11.022876 23124 hybrid_clock.cc:648] HybridClock initialized: now 1790614391022794 us; error 66 us; skew 500 ppm
I20260928 16:53:11.023365 23124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:11 dist-test-slave-wzcb krb5kdc[23111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614391, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:11.027237 23124 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:53:11.029210 23124 webserver.cc:460] Webserver started at http://127.9.93.62:43907/ using document root <none> and password file <none>
I20260928 16:53:11.030066 23124 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:11.030207 23124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:53:11.030547 23124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:53:11.034370 23124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "061d8cab3f3948218138195770bb08e6"
format_stamp: "Formatted at 2026-09-28 16:53:11 on dist-test-slave-wzcb"
server_key: "95fc963790b9c6414988b22afa7dc6e4"
server_key_iv: "0e300554e4fc5ed4c4d6375085d832e8"
server_key_version: "encryptionkey@0"
I20260928 16:53:11.035452 23124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "061d8cab3f3948218138195770bb08e6"
format_stamp: "Formatted at 2026-09-28 16:53:11 on dist-test-slave-wzcb"
server_key: "95fc963790b9c6414988b22afa7dc6e4"
server_key_iv: "0e300554e4fc5ed4c4d6375085d832e8"
server_key_version: "encryptionkey@0"
I20260928 16:53:11.042712 23124 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260928 16:53:11.047745 23139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:11.049649 23124 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260928 16:53:11.049909 23124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "061d8cab3f3948218138195770bb08e6"
format_stamp: "Formatted at 2026-09-28 16:53:11 on dist-test-slave-wzcb"
server_key: "95fc963790b9c6414988b22afa7dc6e4"
server_key_iv: "0e300554e4fc5ed4c4d6375085d832e8"
server_key_version: "encryptionkey@0"
I20260928 16:53:11.050161 23124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:11.066927 23124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:11.072611 23124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:11.072966 23124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:11.093380 23124 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:45901
I20260928 16:53:11.093418 23190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:45901 every 8 connection(s)
I20260928 16:53:11.095032 23124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:53:11.100104 23191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:53:11.100838 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 23124
I20260928 16:53:11.101240 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:53:11.102202 9588 external_mini_cluster.cc:1428] Setting key bfd6bc1dba93ec6b63a29800d057ecce
I20260928 16:53:11.111414 23191 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45901 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.93.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.61@KRBTEST.COM" created.
Sep 28 16:53:11 dist-test-slave-wzcb krb5kdc[23111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614391, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:53:11.142215 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.116044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37674 (local address 127.9.93.62:45901)
0928 16:53:11.117047 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:11.117070 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:11.117528 (+ 458us) server_negotiation.cc:407] Connection header received
0928 16:53:11.119910 (+ 2382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:11.119917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:11.120242 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:11.120465 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:11.122023 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:11.123379 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:11.125976 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:11.126752 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:11.127019 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:11.132755 (+ 5736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:11.132769 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:11.132791 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:11.132845 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:11.137017 (+ 4172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:11.138007 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:11.138018 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:11.138021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:11.138252 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:11.138703 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:11.138707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:11.138710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:11.139547 (+ 837us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:11.139772 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:11.140484 (+ 712us) server_negotiation.cc:299] Negotiation successful
0928 16:53:11.140865 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":708,"thread_start_us":443,"threads_started":1}
I20260928 16:53:11.142256 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.116044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:45901 (local address 127.0.0.1:37674)
0928 16:53:11.117352 (+ 1308us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.117392 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:11.119430 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:11.120521 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:11.120529 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:11.121364 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:53:11.121784 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:11.121807 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:11.123609 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:11.123616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:11.125717 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:11.125748 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:11.127107 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:11.127119 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:11.127382 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:11.128162 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:53:11.128196 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:53:11.132399 (+ 4203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:53:11.137240 (+ 4841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:11.137255 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:11.137267 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:11.137833 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:11.138394 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:11.138398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:11.138400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:11.138595 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:11.139737 (+ 1142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:53:11.139744 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:53:11.140022 (+ 278us) client_negotiation.cc:769] Sending connection context
0928 16:53:11.140315 (+ 293us) client_negotiation.cc:240] Negotiation successful
0928 16:53:11.141155 (+ 840us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1070,"mutex_wait_us":215,"spinlock_wait_cycles":21248,"thread_start_us":846,"threads_started":1}
I20260928 16:53:11.146027 23191 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:53:11.147653 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.147025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:44891 (local address 127.0.0.1:52480)
0928 16:53:11.147267 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.147506 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260928 16:53:11.148046 23143 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.93.61:44891: connect: Connection refused (error 111)
I20260928 16:53:11.150082 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.149627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:44891 (local address 127.0.0.1:52496)
0928 16:53:11.149816 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.149906 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260928 16:53:11.150378 23191 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.61:44891: Network error: Client connection negotiation failed: client connection to 127.9.93.61:44891: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.61.keytab.
Entry for principal kudu/127.9.93.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.61.keytab.
I20260928 16:53:11.166836 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.61:44891
--webserver_interface=127.9.93.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.61.keytab
--principal=kudu/127.9.93.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:35745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:45901,127.9.93.61:44891,127.9.93.60:37545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:53:11.199869 23191 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } attempt: 1
I20260928 16:53:11.201520 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.200841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:44891 (local address 127.0.0.1:52512)
0928 16:53:11.201160 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.201311 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 16:53:11.202967 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.202380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:44891 (local address 127.0.0.1:52516)
0928 16:53:11.202625 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.202752 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 16:53:11.203575 23191 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.61:44891: Network error: Client connection negotiation failed: client connection to 127.9.93.61:44891: connect: Connection refused (error 111)
I20260928 16:53:11.271930 23191 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } attempt: 2
I20260928 16:53:11.273285 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.272740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:44891 (local address 127.0.0.1:52520)
0928 16:53:11.273024 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.273133 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 16:53:11.274642 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.274138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:44891 (local address 127.0.0.1:52536)
0928 16:53:11.274369 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.274455 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260928 16:53:11.274854 23191 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.61:44891: Network error: Client connection negotiation failed: client connection to 127.9.93.61:44891: connect: Connection refused (error 111)
I20260928 16:53:11.366149 23191 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } attempt: 3
I20260928 16:53:11.367518 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.366922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:44891 (local address 127.0.0.1:52540)
0928 16:53:11.367152 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.367258 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 16:53:11.368871 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.368407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:44891 (local address 127.0.0.1:52546)
0928 16:53:11.368601 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.368708 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260928 16:53:11.369074 23191 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.61:44891: Network error: Client connection negotiation failed: client connection to 127.9.93.61:44891: connect: Connection refused (error 111)
I20260928 16:53:11.512339 23191 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } attempt: 4
I20260928 16:53:11.513849 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.513168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:44891 (local address 127.0.0.1:52554)
0928 16:53:11.513577 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.513691 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257}
I20260928 16:53:11.515105 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.514684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:44891 (local address 127.0.0.1:52566)
0928 16:53:11.514865 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.514961 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260928 16:53:11.515327 23191 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.61:44891: Network error: Client connection negotiation failed: client connection to 127.9.93.61:44891: connect: Connection refused (error 111)
W20260928 16:53:11.538707 23198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:11.539088 23198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:11.539239 23198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:11.547086 23198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:53:11.547235 23198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:11.547349 23198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:11.547435 23198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:53:11.547503 23198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:53:11.558343 23198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:35745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:45901,127.9.93.61:44891,127.9.93.60:37545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.61.keytab
--principal=kudu/127.9.93.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.61:44891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.93.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:11.559978 23198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:11.561414 23198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:11.574138 23206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:11.574209 23203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:11.574838 23198 server_base.cc:1034] running on GCE node
W20260928 16:53:11.574909 23204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:11.576126 23198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:11.577842 23198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:11.579077 23198 hybrid_clock.cc:648] HybridClock initialized: now 1790614391579006 us; error 67 us; skew 500 ppm
I20260928 16:53:11.579512 23198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:11 dist-test-slave-wzcb krb5kdc[23111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614391, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:11.582916 23198 init.cc:377] Logged in from keytab as kudu/127.9.93.61@KRBTEST.COM (short username kudu)
I20260928 16:53:11.584621 23198 webserver.cc:460] Webserver started at http://127.9.93.61:37751/ using document root <none> and password file <none>
I20260928 16:53:11.585412 23198 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:11.585533 23198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:53:11.585863 23198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:53:11.589282 23198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-1/data/instance:
uuid: "4d521a45079a408a91b4c6d7359ee555"
format_stamp: "Formatted at 2026-09-28 16:53:11 on dist-test-slave-wzcb"
server_key: "513fbcab310a31ae5835705be209e669"
server_key_iv: "a3b5636bc286da3ba9e5d252e3337b6f"
server_key_version: "encryptionkey@0"
I20260928 16:53:11.590204 23198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-1/wal/instance:
uuid: "4d521a45079a408a91b4c6d7359ee555"
format_stamp: "Formatted at 2026-09-28 16:53:11 on dist-test-slave-wzcb"
server_key: "513fbcab310a31ae5835705be209e669"
server_key_iv: "a3b5636bc286da3ba9e5d252e3337b6f"
server_key_version: "encryptionkey@0"
I20260928 16:53:11.596146 23198 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 16:53:11.600265 23213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:11.602087 23198 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:53:11.602283 23198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-1/wal
uuid: "4d521a45079a408a91b4c6d7359ee555"
format_stamp: "Formatted at 2026-09-28 16:53:11 on dist-test-slave-wzcb"
server_key: "513fbcab310a31ae5835705be209e669"
server_key_iv: "a3b5636bc286da3ba9e5d252e3337b6f"
server_key_version: "encryptionkey@0"
I20260928 16:53:11.602484 23198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:11.640754 23198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:11.645987 23198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:11.646279 23198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:11.666415 23198 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.61:44891
I20260928 16:53:11.666500 23264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.61:44891 every 8 connection(s)
I20260928 16:53:11.668023 23198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-1/data/info.pb
I20260928 16:53:11.672500 23265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:53:11.674539 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 23198
I20260928 16:53:11.674751 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-1/wal/instance
I20260928 16:53:11.675237 9588 external_mini_cluster.cc:1428] Setting key 7b1596811b201b84721f5a71c823cc43
I20260928 16:53:11.679967 23265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45901 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:53:11 dist-test-slave-wzcb krb5kdc[23111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614391, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.61@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.93.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.60@KRBTEST.COM" created.
I20260928 16:53:11.705796 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.682953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37678 (local address 127.9.93.62:45901)
0928 16:53:11.683564 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:11.683572 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:11.684750 (+ 1178us) server_negotiation.cc:407] Connection header received
0928 16:53:11.686875 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:11.686882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:11.686981 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:11.687119 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:11.688921 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:11.690042 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:11.691994 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:11.692655 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:11.692781 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:11.697941 (+ 5160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:11.697955 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:11.697959 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:11.698024 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:11.701358 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:11.702585 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:11.702592 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:11.702595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:11.702781 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:11.703360 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:11.703366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:11.703369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:11.703808 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:11.703993 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:11.705043 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0928 16:53:11.705359 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":265,"threads_started":1}
I20260928 16:53:11.706934 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.683636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:45901 (local address 127.0.0.1:37678)
0928 16:53:11.684534 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.684588 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:11.686545 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:11.687456 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:11.687465 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:11.688041 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:53:11.688706 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:11.688726 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:11.690265 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:11.690270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:11.691758 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:11.691775 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:11.692954 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:11.692963 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:11.693326 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:11.694200 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:53:11.694237 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:53:11.697543 (+ 3306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:53:11.701606 (+ 4063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:11.701616 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:11.701635 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:11.702378 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:11.702896 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:11.702901 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:11.702904 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:11.703178 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:11.703962 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:53:11.703971 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:53:11.704509 (+ 538us) client_negotiation.cc:769] Sending connection context
0928 16:53:11.704853 (+ 344us) client_negotiation.cc:240] Negotiation successful
0928 16:53:11.705482 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":379,"threads_started":1}
I20260928 16:53:11.709371 23265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:53:11 dist-test-slave-wzcb krb5kdc[23111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614391, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.61@KRBTEST.COM for kudu/127.9.93.61@KRBTEST.COM
I20260928 16:53:11.726511 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.710244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:44891 (local address 127.0.0.1:52582)
0928 16:53:11.710463 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.710484 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:11.710625 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:11.711581 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:11.711586 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:11.712105 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:53:11.712308 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:11.712320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:11.713915 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:11.713920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:11.715167 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:11.715185 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:11.716270 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:11.716276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:11.716533 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:11.717187 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:53:11.717210 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:53:11.719940 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:53:11.723595 (+ 3655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:11.723602 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:11.723607 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:11.724191 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:11.724704 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:11.724709 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:11.724713 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:11.724919 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:11.725649 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:53:11.725653 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:53:11.725788 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 16:53:11.725911 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 16:53:11.726174 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 16:53:11.726975 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.710360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52582 (local address 127.9.93.61:44891)
0928 16:53:11.711019 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:11.711026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:11.711055 (+ 29us) server_negotiation.cc:407] Connection header received
0928 16:53:11.711107 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:11.711113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:11.711409 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:11.711542 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:11.712548 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:11.713734 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:11.715425 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:11.716089 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:11.716206 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:11.720240 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:11.720260 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:11.720277 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:11.720317 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:11.723419 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:11.724362 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:11.724401 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:11.724403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:11.724546 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:11.725035 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:11.725039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:11.725041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:11.725501 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:11.725667 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:11.726174 (+ 507us) server_negotiation.cc:299] Negotiation successful
0928 16:53:11.726461 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":352,"threads_started":1}
I20260928 16:53:11.729800 23265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:53:11.731247 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.730662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:37545 (local address 127.0.0.1:58318)
0928 16:53:11.730897 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.731080 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260928 16:53:11.731666 23216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.60.keytab.
Entry for principal kudu/127.9.93.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.60.keytab.
I20260928 16:53:11.733635 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.733170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:37545 (local address 127.0.0.1:58328)
0928 16:53:11.733379 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.733486 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 16:53:11.733882 23265 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:37545: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
I20260928 16:53:11.734506 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.60:37545
--webserver_interface=127.9.93.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.60.keytab
--principal=kudu/127.9.93.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:35745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:45901,127.9.93.61:44891,127.9.93.60:37545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:53:11.783247 23265 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } attempt: 1
I20260928 16:53:11.784539 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.784044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:37545 (local address 127.0.0.1:58342)
0928 16:53:11.784243 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.784363 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 16:53:11.785727 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.785365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:37545 (local address 127.0.0.1:58352)
0928 16:53:11.785531 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.785609 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 16:53:11.785965 23265 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:37545: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
I20260928 16:53:11.814659 23191 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } attempt: 5
Sep 28 16:53:11 dist-test-slave-wzcb krb5kdc[23111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614391, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for kudu/127.9.93.61@KRBTEST.COM
I20260928 16:53:11.830852 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.815679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52596 (local address 127.9.93.61:44891)
0928 16:53:11.815876 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:11.815893 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:11.815917 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:53:11.816188 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:11.816193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:11.816288 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:11.816395 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:11.817546 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:11.818326 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:11.820192 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:11.820710 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:11.820817 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:11.824955 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:11.824962 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:11.824965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:11.825009 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:11.827787 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:11.828722 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:11.828728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:11.828730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:11.828885 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:11.829358 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:11.829362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:11.829364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:11.829718 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:11.829871 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:11.830363 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 16:53:11.830570 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 16:53:11.830973 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.815581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:44891 (local address 127.0.0.1:52596)
0928 16:53:11.815827 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.815853 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:11.816011 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:11.816468 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:11.816474 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:11.817071 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:53:11.817337 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:11.817352 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:11.818512 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:11.818519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:11.820001 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:11.820023 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:11.820895 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:11.820902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:11.821137 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:11.821862 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:53:11.821890 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:53:11.824640 (+ 2750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:53:11.827970 (+ 3330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:11.827976 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:11.827981 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:11.828565 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:11.829008 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:11.829013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:11.829016 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:11.829204 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:11.829846 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:53:11.829852 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:53:11.830050 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 16:53:11.830213 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 16:53:11.830592 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 16:53:11.832387 23191 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:53:11.833756 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.833195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:37545 (local address 127.0.0.1:58366)
0928 16:53:11.833405 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.833523 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 16:53:11.834901 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.834446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:37545 (local address 127.0.0.1:58376)
0928 16:53:11.834655 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.834761 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 16:53:11.835151 23191 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:37545: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
I20260928 16:53:11.854211 23265 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } attempt: 2
I20260928 16:53:11.855639 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.855069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:37545 (local address 127.0.0.1:58380)
0928 16:53:11.855282 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.855458 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 16:53:11.856804 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.856406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:37545 (local address 127.0.0.1:58384)
0928 16:53:11.856567 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.856656 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 16:53:11.857040 23265 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:37545: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
I20260928 16:53:11.886452 23191 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } attempt: 1
I20260928 16:53:11.888018 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.887425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:37545 (local address 127.0.0.1:58394)
0928 16:53:11.887672 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.887795 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 16:53:11.889235 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.888799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:37545 (local address 127.0.0.1:58398)
0928 16:53:11.888975 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.889085 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 16:53:11.889529 23191 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:37545: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
I20260928 16:53:11.948338 23265 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } attempt: 3
I20260928 16:53:11.949777 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.949179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:37545 (local address 127.0.0.1:58406)
0928 16:53:11.949483 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.949589 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 16:53:11.951015 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.950521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:37545 (local address 127.0.0.1:58416)
0928 16:53:11.950798 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.950888 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
W20260928 16:53:11.951243 23265 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:37545: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
I20260928 16:53:11.957799 23191 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } attempt: 2
I20260928 16:53:11.959399 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.958649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:37545 (local address 127.0.0.1:58426)
0928 16:53:11.958939 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.959092 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 16:53:11.960779 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:11.960188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:37545 (local address 127.0.0.1:58430)
0928 16:53:11.960426 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:11.960552 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260928 16:53:11.961058 23191 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:37545: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
I20260928 16:53:12.067386 23191 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } attempt: 3
I20260928 16:53:12.068802 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:12.068203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:37545 (local address 127.0.0.1:58442)
0928 16:53:12.068464 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:12.068586 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 16:53:12.070154 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:12.069624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:37545 (local address 127.0.0.1:58448)
0928 16:53:12.069868 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:12.069992 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260928 16:53:12.070480 23191 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:37545: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
I20260928 16:53:12.094497 23265 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } attempt: 4
I20260928 16:53:12.096006 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:12.095235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:37545 (local address 127.0.0.1:58458)
0928 16:53:12.095653 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:12.095772 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229}
I20260928 16:53:12.097258 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:12.096790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:37545 (local address 127.0.0.1:58470)
0928 16:53:12.097007 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:12.097100 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"threads_started":1}
W20260928 16:53:12.097893 23265 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:37545: Network error: Client connection negotiation failed: client connection to 127.9.93.60:37545: connect: Connection refused (error 111)
W20260928 16:53:12.104697 23273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:12.105048 23273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:12.105185 23273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:12.113106 23273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:53:12.113256 23273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:12.113341 23273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:12.113425 23273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:53:12.113490 23273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:53:12.124189 23273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:35745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:45901,127.9.93.61:44891,127.9.93.60:37545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.60.keytab
--principal=kudu/127.9.93.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.60:37545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.93.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:12.125789 23273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:12.127333 23273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:12.140347 23280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:12.140714 23279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:12.140779 23282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:12.141312 23273 server_base.cc:1034] running on GCE node
I20260928 16:53:12.142223 23273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:12.144063 23273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:12.145292 23273 hybrid_clock.cc:648] HybridClock initialized: now 1790614392145223 us; error 64 us; skew 500 ppm
I20260928 16:53:12.145711 23273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:12 dist-test-slave-wzcb krb5kdc[23111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614392, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:12.149252 23273 init.cc:377] Logged in from keytab as kudu/127.9.93.60@KRBTEST.COM (short username kudu)
I20260928 16:53:12.151144 23273 webserver.cc:460] Webserver started at http://127.9.93.60:39275/ using document root <none> and password file <none>
I20260928 16:53:12.152108 23273 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:12.152251 23273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:53:12.152590 23273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:53:12.156243 23273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-2/data/instance:
uuid: "a3a2a4e2431646ccb45c9a521a5b5255"
format_stamp: "Formatted at 2026-09-28 16:53:12 on dist-test-slave-wzcb"
server_key: "08e278555240637c9e12a8af952d82b1"
server_key_iv: "866bc728cc661858ee97d1eb82dfd950"
server_key_version: "encryptionkey@0"
I20260928 16:53:12.157176 23273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-2/wal/instance:
uuid: "a3a2a4e2431646ccb45c9a521a5b5255"
format_stamp: "Formatted at 2026-09-28 16:53:12 on dist-test-slave-wzcb"
server_key: "08e278555240637c9e12a8af952d82b1"
server_key_iv: "866bc728cc661858ee97d1eb82dfd950"
server_key_version: "encryptionkey@0"
I20260928 16:53:12.163183 23273 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 16:53:12.167286 23289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:12.168959 23273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:53:12.169158 23273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-2/wal
uuid: "a3a2a4e2431646ccb45c9a521a5b5255"
format_stamp: "Formatted at 2026-09-28 16:53:12 on dist-test-slave-wzcb"
server_key: "08e278555240637c9e12a8af952d82b1"
server_key_iv: "866bc728cc661858ee97d1eb82dfd950"
server_key_version: "encryptionkey@0"
I20260928 16:53:12.169382 23273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:12.183684 23273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:12.188128 23273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:12.188437 23273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:12.207439 23273 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.60:37545
I20260928 16:53:12.207440 23340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.60:37545 every 8 connection(s)
I20260928 16:53:12.209023 23273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-2/data/info.pb
I20260928 16:53:12.210544 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 23273
I20260928 16:53:12.210723 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790614166338986-9588-0/minicluster-data/master-2/wal/instance
I20260928 16:53:12.211247 9588 external_mini_cluster.cc:1428] Setting key 22c8527f786a4956b4388285bf07a89b
I20260928 16:53:12.214394 23341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:53:12.222087 23341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45901 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:53:12 dist-test-slave-wzcb krb5kdc[23111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.60@KRBTEST.COM
Sep 28 16:53:12 dist-test-slave-wzcb krb5kdc[23111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614392, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.60@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:53:12.237486 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:12.213467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58482 (local address 127.9.93.60:37545)
0928 16:53:12.214381 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:12.214398 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:12.214448 (+ 50us) server_negotiation.cc:407] Connection header received
0928 16:53:12.216258 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:12.216285 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:12.216742 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:12.217188 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:12.218930 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.220437 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.221854 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.222602 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.222804 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:12.228105 (+ 5301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:12.228116 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:12.228132 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:12.228181 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:12.231779 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:12.232825 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:12.232833 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:12.232840 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:12.233047 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:12.234037 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:12.234042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:12.234044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:12.234663 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:12.234969 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:12.235429 (+ 460us) server_negotiation.cc:299] Negotiation successful
0928 16:53:12.235918 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":433,"threads_started":1}
I20260928 16:53:12.241305 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:12.225161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:45901 (local address 127.0.0.1:37690)
0928 16:53:12.225965 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:12.226022 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:12.226190 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:12.226654 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:12.226661 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:12.227569 (+ 908us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:53:12.227977 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:12.227992 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.229327 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.229331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:12.230488 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:12.230511 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.231522 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.231528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:12.231663 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:12.232243 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:53:12.232268 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:53:12.234566 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:53:12.238360 (+ 3794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:12.238370 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:12.238377 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:12.238962 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:12.239425 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:12.239428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:12.239430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:12.239572 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:12.240179 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:53:12.240188 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:53:12.240431 (+ 243us) client_negotiation.cc:769] Sending connection context
0928 16:53:12.240611 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 16:53:12.240916 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":450,"threads_started":1}
I20260928 16:53:12.241343 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:12.225132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37690 (local address 127.9.93.62:45901)
0928 16:53:12.225633 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:12.225640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:12.226136 (+ 496us) server_negotiation.cc:407] Connection header received
0928 16:53:12.226351 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:12.226357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:12.226449 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:12.226544 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:12.228161 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.229167 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.230651 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.231291 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.231467 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:12.234784 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:12.234792 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:12.234797 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:12.234857 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:12.238231 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:12.239083 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:12.239089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:12.239092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:12.239256 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:12.239676 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:12.239680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:12.239684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:12.240037 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:12.240173 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:12.240749 (+ 576us) server_negotiation.cc:299] Negotiation successful
0928 16:53:12.240949 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":234,"threads_started":1}
I20260928 16:53:12.243165 23341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:53:12.247710 23191 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } attempt: 4
Sep 28 16:53:12 dist-test-slave-wzcb krb5kdc[23111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
Sep 28 16:53:12 dist-test-slave-wzcb krb5kdc[23111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614392, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.60@KRBTEST.COM for kudu/127.9.93.61@KRBTEST.COM
Sep 28 16:53:12 dist-test-slave-wzcb krb5kdc[23111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614391, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for kudu/127.9.93.60@KRBTEST.COM
I20260928 16:53:12.258291 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:12.243830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:44891 (local address 127.0.0.1:52598)
0928 16:53:12.244002 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:12.244021 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:12.244114 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:12.244406 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:12.244410 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:12.244842 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:53:12.244997 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:12.245006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.246352 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.246357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:12.247623 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:12.247637 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.248673 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.248679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:12.248863 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:12.249437 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:53:12.249459 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:53:12.252245 (+ 2786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:53:12.255774 (+ 3529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:12.255779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:12.255784 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:12.256227 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:12.256600 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:12.256606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:12.256609 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:12.256784 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:12.257380 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:53:12.257389 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:53:12.257559 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 16:53:12.257708 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 16:53:12.257963 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 16:53:12.259080 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:12.241144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37696 (local address 127.9.93.62:45901)
0928 16:53:12.241585 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:12.241591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:12.241610 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:53:12.241663 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:12.241668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:12.241782 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:12.241899 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:12.243653 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.244622 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.246342 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.247060 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.247188 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:12.252235 (+ 5047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:12.252241 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:12.252247 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:12.252285 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:12.255510 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:12.256436 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:12.256442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:12.256445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:12.256597 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:12.257041 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:12.257047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:12.257050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:12.257405 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:12.257684 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:12.258271 (+ 587us) server_negotiation.cc:299] Negotiation successful
0928 16:53:12.258747 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":596,"threads_started":1}
I20260928 16:53:12.259294 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:12.243986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52598 (local address 127.9.93.61:44891)
0928 16:53:12.244147 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:12.244151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:12.244181 (+ 30us) server_negotiation.cc:407] Connection header received
0928 16:53:12.244230 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:12.244234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:12.244294 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:12.244364 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:12.245160 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.246186 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.247812 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.248393 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.248516 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:12.252433 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:12.252438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:12.252441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:12.252478 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:12.255648 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:12.256343 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:12.256350 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:12.256354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:12.256487 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:12.256922 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:12.256927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:12.256929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:12.257263 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:12.257407 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:12.258702 (+ 1295us) server_negotiation.cc:299] Negotiation successful
0928 16:53:12.258951 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:53:12.260473 23341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:53:12.262990 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:12.248297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:37545 (local address 127.0.0.1:58490)
0928 16:53:12.248565 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:12.248585 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:12.248691 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:12.249105 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:12.249110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:12.249551 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:53:12.249752 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:12.249763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.250977 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.250980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:12.251995 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:12.252007 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.252902 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.252908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:12.253103 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:12.253649 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:53:12.253672 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:53:12.256695 (+ 3023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:53:12.260311 (+ 3616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:12.260317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:12.260321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:12.260788 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:12.261251 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:12.261256 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:12.261259 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:12.261430 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:12.262183 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:53:12.262187 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:53:12.262327 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 16:53:12.262418 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 16:53:12.262641 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260928 16:53:12.264086 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:12.248374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58490 (local address 127.9.93.60:37545)
0928 16:53:12.248577 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:12.248583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:12.248795 (+ 212us) server_negotiation.cc:407] Connection header received
0928 16:53:12.248853 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:12.248858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:12.248932 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:12.249038 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:12.249905 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.250852 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.252123 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.252724 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.252850 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:12.256845 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:12.256850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:12.256855 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:12.256890 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:12.260063 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:12.260898 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:12.260903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:12.260907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:12.261097 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:12.261585 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:12.261589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:12.261593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:12.261991 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:12.263515 (+ 1524us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:12.263634 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 16:53:12.263848 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 16:53:12 dist-test-slave-wzcb krb5kdc[23111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614392, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.60@KRBTEST.COM for kudu/127.9.93.60@KRBTEST.COM
I20260928 16:53:12.270112 23191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6: Bootstrap starting.
Sep 28 16:53:12 dist-test-slave-wzcb krb5kdc[23111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.61@KRBTEST.COM
I20260928 16:53:12.274773 23191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6: Neither blocks nor log segments found. Creating new log.
I20260928 16:53:12.276290 23191 log.cc:824] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6: Log is configured to *not* fsync() on all Append() calls
I20260928 16:53:12.277087 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:12.261274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:37545 (local address 127.0.0.1:58500)
0928 16:53:12.261473 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:12.261491 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:12.261579 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:12.262756 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:12.262761 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:12.263218 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:53:12.263461 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:12.263473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.265151 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.265156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:12.266071 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:12.266083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.266998 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.267005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:12.267222 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:12.267830 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:53:12.267852 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:53:12.270513 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:53:12.274243 (+ 3730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:12.274249 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:12.274254 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:12.274812 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:12.275240 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:12.275244 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:12.275248 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:12.275440 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:12.276051 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:53:12.276056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:53:12.276215 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 16:53:12.276373 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 16:53:12.276697 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 16:53:12.277910 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:12.261725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58500 (local address 127.9.93.60:37545)
0928 16:53:12.262452 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:12.262457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:12.262479 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:53:12.262522 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:12.262527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:12.262595 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:12.262705 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:12.263652 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.265037 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.266209 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.266820 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.266961 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:12.270798 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:12.270804 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:12.270809 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:12.270843 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:12.274092 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:12.274952 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:12.274957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:12.274960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:12.275108 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:12.275574 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:12.275579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:12.275582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:12.275899 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:12.276074 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:12.277469 (+ 1395us) server_negotiation.cc:299] Negotiation successful
0928 16:53:12.277638 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":380,"threads_started":1}
I20260928 16:53:12.278136 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:12.261304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52606 (local address 127.9.93.61:44891)
0928 16:53:12.261469 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:12.261475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:12.261492 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:53:12.261541 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:12.261546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:12.261614 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:12.261700 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:12.263411 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.264444 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.266197 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.266848 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.266964 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:12.271828 (+ 4864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:12.271833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:12.271836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:12.271882 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:12.274802 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:12.275674 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:12.275680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:12.275683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:12.275850 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:12.276302 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:12.276307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:12.276311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:12.276638 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:12.276939 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:12.277565 (+ 626us) server_negotiation.cc:299] Negotiation successful
0928 16:53:12.277798 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:53:12.280004 23191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6: No bootstrap required, opened a new log
I20260928 16:53:12.283368 23341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3a2a4e2431646ccb45c9a521a5b5255: Bootstrap starting.
I20260928 16:53:12.284590 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:53:12.286821 23341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3a2a4e2431646ccb45c9a521a5b5255: Neither blocks nor log segments found. Creating new log.
I20260928 16:53:12.288071 23341 log.cc:824] T 00000000000000000000000000000000 P a3a2a4e2431646ccb45c9a521a5b5255: Log is configured to *not* fsync() on all Append() calls
I20260928 16:53:12.290668 23191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061d8cab3f3948218138195770bb08e6" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45901 } } peers { permanent_uuid: "4d521a45079a408a91b4c6d7359ee555" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } } peers { permanent_uuid: "a3a2a4e2431646ccb45c9a521a5b5255" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } }
I20260928 16:53:12.291100 23191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:53:12.291235 23341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3a2a4e2431646ccb45c9a521a5b5255: No bootstrap required, opened a new log
I20260928 16:53:12.291342 23191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 061d8cab3f3948218138195770bb08e6, State: Initialized, Role: FOLLOWER
I20260928 16:53:12.292032 23191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061d8cab3f3948218138195770bb08e6" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45901 } } peers { permanent_uuid: "4d521a45079a408a91b4c6d7359ee555" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } } peers { permanent_uuid: "a3a2a4e2431646ccb45c9a521a5b5255" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } }
I20260928 16:53:12.293027 23356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061d8cab3f3948218138195770bb08e6" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45901 } } peers { permanent_uuid: "4d521a45079a408a91b4c6d7359ee555" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } } peers { permanent_uuid: "a3a2a4e2431646ccb45c9a521a5b5255" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } } }
I20260928 16:53:12.293514 23356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:53:12.294305 23191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:53:12.303982 23341 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3a2a4e2431646ccb45c9a521a5b5255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061d8cab3f3948218138195770bb08e6" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45901 } } peers { permanent_uuid: "4d521a45079a408a91b4c6d7359ee555" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } } peers { permanent_uuid: "a3a2a4e2431646ccb45c9a521a5b5255" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } }
I20260928 16:53:12.304658 23341 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3a2a4e2431646ccb45c9a521a5b5255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:53:12.304855 23341 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3a2a4e2431646ccb45c9a521a5b5255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3a2a4e2431646ccb45c9a521a5b5255, State: Initialized, Role: FOLLOWER
I20260928 16:53:12.305770 23341 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3a2a4e2431646ccb45c9a521a5b5255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061d8cab3f3948218138195770bb08e6" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45901 } } peers { permanent_uuid: "4d521a45079a408a91b4c6d7359ee555" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } } peers { permanent_uuid: "a3a2a4e2431646ccb45c9a521a5b5255" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } }
I20260928 16:53:12.307541 23365 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a2a4e2431646ccb45c9a521a5b5255 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061d8cab3f3948218138195770bb08e6" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45901 } } peers { permanent_uuid: "4d521a45079a408a91b4c6d7359ee555" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } } peers { permanent_uuid: "a3a2a4e2431646ccb45c9a521a5b5255" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } } }
I20260928 16:53:12.308128 23365 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a2a4e2431646ccb45c9a521a5b5255 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:53:12.309203 23341 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3a2a4e2431646ccb45c9a521a5b5255 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:53:12.323252 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:12.291834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59648 (local address 127.9.93.60:37545)
0928 16:53:12.292107 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:12.292114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:12.301129 (+ 9015us) server_negotiation.cc:407] Connection header received
0928 16:53:12.301425 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:12.301431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:12.301540 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:12.301618 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:12.304230 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.305282 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.307808 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.308512 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.308622 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:12.313543 (+ 4921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:12.313550 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:12.313556 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:12.313599 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:12.317494 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:12.318496 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:12.318503 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:12.318519 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:12.318672 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:12.319166 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:12.319172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:12.319175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:12.319613 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:12.319871 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:12.322650 (+ 2779us) server_negotiation.cc:299] Negotiation successful
0928 16:53:12.322905 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 16:53:12.330932 23380 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3a2a4e2431646ccb45c9a521a5b5255: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:53:12.331120 23380 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 16:53:12.331499 23145 tablet.cc:2369] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:53:12.332562 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:12.300166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42908 (local address 127.9.93.61:44891)
0928 16:53:12.300366 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:12.300372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:12.304491 (+ 4119us) server_negotiation.cc:407] Connection header received
0928 16:53:12.304764 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:12.304770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:12.304864 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:12.304934 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:12.307322 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.308374 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.310041 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.310700 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.310822 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:12.320469 (+ 9647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:12.320478 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:12.320484 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:12.320548 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:12.326547 (+ 5999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:12.329381 (+ 2834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:12.329388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:12.329392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:12.329593 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:12.330078 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:12.330083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:12.330086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:12.330493 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:12.330771 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:12.331942 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0928 16:53:12.332202 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 16:53:12.340713 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:12.292593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48008 (local address 127.9.93.62:45901)
0928 16:53:12.295800 (+ 3207us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:12.295807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:12.304007 (+ 8200us) server_negotiation.cc:407] Connection header received
0928 16:53:12.305286 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:12.305292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:12.305376 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:12.305445 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:12.311680 (+ 6235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.312784 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.314692 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.315410 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.315519 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:12.322616 (+ 7097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:12.322623 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:12.322628 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:12.322689 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:12.326919 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:12.331372 (+ 4453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:12.331378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:12.331382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:12.333212 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:12.337586 (+ 4374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:12.337597 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:12.337601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:12.338065 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:12.338334 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:12.339621 (+ 1287us) server_negotiation.cc:299] Negotiation successful
0928 16:53:12.339899 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3053}
I20260928 16:53:12.335573 23124 master_runner.cc:186] Error getting master registration for 127.9.93.61:44891: OK, instance_id { permanent_uuid: "4d521a45079a408a91b4c6d7359ee555" instance_seqno: 1790614391660799 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:53:12.349311 23124 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 16:53:12.349058 23381 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:53:12.349464 23381 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:53:12.341048 23273 master_runner.cc:186] Error getting master registration for 127.9.93.61:44891: OK, instance_id { permanent_uuid: "4d521a45079a408a91b4c6d7359ee555" instance_seqno: 1790614391660799 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:53:12.352142 23273 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:53:12.397140 23265 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } attempt: 5
Sep 28 16:53:12 dist-test-slave-wzcb krb5kdc[23111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614391, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.61@KRBTEST.COM for kudu/127.9.93.60@KRBTEST.COM
I20260928 16:53:12.421768 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:12.398029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59658 (local address 127.9.93.60:37545)
0928 16:53:12.398235 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:12.398242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:12.398769 (+ 527us) server_negotiation.cc:407] Connection header received
0928 16:53:12.399284 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:12.399290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:12.399477 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:12.399654 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:12.400963 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.402195 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.404166 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.404992 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.405130 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:12.412248 (+ 7118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:12.412256 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:12.412261 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:12.412305 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:12.416456 (+ 4151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:12.417409 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:12.417416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:12.417420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:12.417633 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:12.418101 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:12.418106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:12.418110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:12.418546 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:12.418739 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:12.419194 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 16:53:12.419485 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:53:12.422452 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:12.398400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:37545 (local address 127.0.0.1:59658)
0928 16:53:12.398646 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:12.398668 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:12.398916 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:12.399870 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:12.399876 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:12.400520 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:53:12.400793 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:12.400807 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.402376 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.402383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:12.404007 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:12.404026 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:12.405190 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:12.405198 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:12.405388 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:12.406143 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:53:12.406170 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:53:12.411960 (+ 5790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:53:12.416653 (+ 4693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:12.416661 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:12.416666 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:12.417265 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:12.417766 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:12.417772 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:12.417775 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:12.417983 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:12.418687 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:53:12.418692 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:53:12.418896 (+ 204us) client_negotiation.cc:769] Sending connection context
0928 16:53:12.419075 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 16:53:12.422034 (+ 2959us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 16:53:12.429096 23265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d521a45079a408a91b4c6d7359ee555: Bootstrap starting.
I20260928 16:53:12.433729 23265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d521a45079a408a91b4c6d7359ee555: Neither blocks nor log segments found. Creating new log.
I20260928 16:53:12.435324 23265 log.cc:824] T 00000000000000000000000000000000 P 4d521a45079a408a91b4c6d7359ee555: Log is configured to *not* fsync() on all Append() calls
W20260928 16:53:12.440815 23295 tablet.cc:2369] T 00000000000000000000000000000000 P a3a2a4e2431646ccb45c9a521a5b5255: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:53:12.441576 23265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d521a45079a408a91b4c6d7359ee555: No bootstrap required, opened a new log
I20260928 16:53:12.454583 23265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d521a45079a408a91b4c6d7359ee555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061d8cab3f3948218138195770bb08e6" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45901 } } peers { permanent_uuid: "4d521a45079a408a91b4c6d7359ee555" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } } peers { permanent_uuid: "a3a2a4e2431646ccb45c9a521a5b5255" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } }
I20260928 16:53:12.455196 23265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d521a45079a408a91b4c6d7359ee555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:53:12.455399 23265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d521a45079a408a91b4c6d7359ee555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d521a45079a408a91b4c6d7359ee555, State: Initialized, Role: FOLLOWER
I20260928 16:53:12.456291 23265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d521a45079a408a91b4c6d7359ee555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061d8cab3f3948218138195770bb08e6" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45901 } } peers { permanent_uuid: "4d521a45079a408a91b4c6d7359ee555" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } } peers { permanent_uuid: "a3a2a4e2431646ccb45c9a521a5b5255" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } }
I20260928 16:53:12.457432 23385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d521a45079a408a91b4c6d7359ee555 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061d8cab3f3948218138195770bb08e6" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45901 } } peers { permanent_uuid: "4d521a45079a408a91b4c6d7359ee555" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } } peers { permanent_uuid: "a3a2a4e2431646ccb45c9a521a5b5255" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } } }
I20260928 16:53:12.457919 23385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d521a45079a408a91b4c6d7359ee555 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:53:12.458988 23265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d521a45079a408a91b4c6d7359ee555 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:53:12.483392 23198 master_runner.cc:263] No leader master found from master 4d521a45079a408a91b4c6d7359ee555
I20260928 16:53:12.483589 23198 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 16:53:12.490774 23396 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d521a45079a408a91b4c6d7359ee555: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:53:12.491006 23396 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:53:12.584523 23356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:53:12.584915 23356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061d8cab3f3948218138195770bb08e6" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45901 } } peers { permanent_uuid: "4d521a45079a408a91b4c6d7359ee555" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } } peers { permanent_uuid: "a3a2a4e2431646ccb45c9a521a5b5255" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } }
I20260928 16:53:12.586814 23356 leader_election.cc:288] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d521a45079a408a91b4c6d7359ee555 (127.9.93.61:44891), a3a2a4e2431646ccb45c9a521a5b5255 (127.9.93.60:37545)
I20260928 16:53:12.588433 23316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "061d8cab3f3948218138195770bb08e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3a2a4e2431646ccb45c9a521a5b5255" is_pre_election: true
I20260928 16:53:12.589175 23316 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a3a2a4e2431646ccb45c9a521a5b5255 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 061d8cab3f3948218138195770bb08e6 in term 0.
I20260928 16:53:12.590613 23240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "061d8cab3f3948218138195770bb08e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d521a45079a408a91b4c6d7359ee555" is_pre_election: true
I20260928 16:53:12.591944 23142 leader_election.cc:302] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 061d8cab3f3948218138195770bb08e6, a3a2a4e2431646ccb45c9a521a5b5255; no voters:
I20260928 16:53:12.592597 23240 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4d521a45079a408a91b4c6d7359ee555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 061d8cab3f3948218138195770bb08e6 in term 0.
I20260928 16:53:12.593945 23356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:53:12.594135 23356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:53:12.594272 23356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:53:12.603107 23356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061d8cab3f3948218138195770bb08e6" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45901 } } peers { permanent_uuid: "4d521a45079a408a91b4c6d7359ee555" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } } peers { permanent_uuid: "a3a2a4e2431646ccb45c9a521a5b5255" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } }
I20260928 16:53:12.607445 23240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "061d8cab3f3948218138195770bb08e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d521a45079a408a91b4c6d7359ee555"
I20260928 16:53:12.607762 23240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d521a45079a408a91b4c6d7359ee555 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:53:12.612108 23240 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4d521a45079a408a91b4c6d7359ee555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 061d8cab3f3948218138195770bb08e6 in term 1.
I20260928 16:53:12.612924 23143 leader_election.cc:302] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 061d8cab3f3948218138195770bb08e6, 4d521a45079a408a91b4c6d7359ee555; no voters:
I20260928 16:53:12.618212 23316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "061d8cab3f3948218138195770bb08e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3a2a4e2431646ccb45c9a521a5b5255"
I20260928 16:53:12.618517 23316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3a2a4e2431646ccb45c9a521a5b5255 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:53:12.622613 23316 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a3a2a4e2431646ccb45c9a521a5b5255 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 061d8cab3f3948218138195770bb08e6 in term 1.
I20260928 16:53:12.623977 23356 leader_election.cc:288] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d521a45079a408a91b4c6d7359ee555 (127.9.93.61:44891), a3a2a4e2431646ccb45c9a521a5b5255 (127.9.93.60:37545)
I20260928 16:53:12.624552 23356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:53:12.629695 23356 raft_consensus.cc:686] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [term 1 LEADER]: Becoming Leader. State: Replica: 061d8cab3f3948218138195770bb08e6, State: Running, Role: LEADER
I20260928 16:53:12.630801 23356 consensus_queue.cc:237] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061d8cab3f3948218138195770bb08e6" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45901 } } peers { permanent_uuid: "4d521a45079a408a91b4c6d7359ee555" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } } peers { permanent_uuid: "a3a2a4e2431646ccb45c9a521a5b5255" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } }
I20260928 16:53:12.643203 23398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 061d8cab3f3948218138195770bb08e6. Latest consensus state: current_term: 1 leader_uuid: "061d8cab3f3948218138195770bb08e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061d8cab3f3948218138195770bb08e6" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45901 } } peers { permanent_uuid: "4d521a45079a408a91b4c6d7359ee555" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } } peers { permanent_uuid: "a3a2a4e2431646ccb45c9a521a5b5255" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } } }
I20260928 16:53:12.644073 23398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [sys.catalog]: This master's current role is: LEADER
W20260928 16:53:12.649541 23219 tablet.cc:2369] T 00000000000000000000000000000000 P 4d521a45079a408a91b4c6d7359ee555: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:53:12.651341 23400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:53:12.654520 23400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:53:12.678960 23316 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a3a2a4e2431646ccb45c9a521a5b5255 [term 1 FOLLOWER]: Refusing update from remote peer 061d8cab3f3948218138195770bb08e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:53:12.680496 23398 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3a2a4e2431646ccb45c9a521a5b5255" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:53:12.688855 23240 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d521a45079a408a91b4c6d7359ee555 [term 1 FOLLOWER]: Refusing update from remote peer 061d8cab3f3948218138195770bb08e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:53:12.690248 23398 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d521a45079a408a91b4c6d7359ee555" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:53:12.738139 23365 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a2a4e2431646ccb45c9a521a5b5255 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 061d8cab3f3948218138195770bb08e6. Latest consensus state: current_term: 1 leader_uuid: "061d8cab3f3948218138195770bb08e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061d8cab3f3948218138195770bb08e6" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45901 } } peers { permanent_uuid: "4d521a45079a408a91b4c6d7359ee555" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } } peers { permanent_uuid: "a3a2a4e2431646ccb45c9a521a5b5255" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } } }
I20260928 16:53:12.738498 23365 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a2a4e2431646ccb45c9a521a5b5255 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:53:12.741997 23385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d521a45079a408a91b4c6d7359ee555 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 061d8cab3f3948218138195770bb08e6. Latest consensus state: current_term: 1 leader_uuid: "061d8cab3f3948218138195770bb08e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061d8cab3f3948218138195770bb08e6" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45901 } } peers { permanent_uuid: "4d521a45079a408a91b4c6d7359ee555" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } } peers { permanent_uuid: "a3a2a4e2431646ccb45c9a521a5b5255" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } } }
I20260928 16:53:12.742308 23385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d521a45079a408a91b4c6d7359ee555 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:53:12.745577 23401 mvcc.cc:205] Tried to move back new op lower bound from 7334356552396574720 to 7334356552230150144. Current Snapshot: MvccSnapshot[applied={T|T < 7334356552396574720}]
I20260928 16:53:12.748421 23403 mvcc.cc:205] Tried to move back new op lower bound from 7334356552396574720 to 7334356552230150144. Current Snapshot: MvccSnapshot[applied={T|T < 7334356552396574720}]
I20260928 16:53:12.760390 23398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "061d8cab3f3948218138195770bb08e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061d8cab3f3948218138195770bb08e6" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45901 } } peers { permanent_uuid: "4d521a45079a408a91b4c6d7359ee555" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } } peers { permanent_uuid: "a3a2a4e2431646ccb45c9a521a5b5255" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } } }
I20260928 16:53:12.760793 23398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [sys.catalog]: This master's current role is: LEADER
I20260928 16:53:12.761471 23398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "061d8cab3f3948218138195770bb08e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061d8cab3f3948218138195770bb08e6" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45901 } } peers { permanent_uuid: "4d521a45079a408a91b4c6d7359ee555" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } } peers { permanent_uuid: "a3a2a4e2431646ccb45c9a521a5b5255" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } } }
I20260928 16:53:12.761709 23398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6 [sys.catalog]: This master's current role is: LEADER
I20260928 16:53:12.767112 23385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d521a45079a408a91b4c6d7359ee555 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "061d8cab3f3948218138195770bb08e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061d8cab3f3948218138195770bb08e6" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45901 } } peers { permanent_uuid: "4d521a45079a408a91b4c6d7359ee555" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } } peers { permanent_uuid: "a3a2a4e2431646ccb45c9a521a5b5255" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } } }
I20260928 16:53:12.767464 23385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d521a45079a408a91b4c6d7359ee555 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:53:12.772446 23365 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a2a4e2431646ccb45c9a521a5b5255 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "061d8cab3f3948218138195770bb08e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061d8cab3f3948218138195770bb08e6" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 45901 } } peers { permanent_uuid: "4d521a45079a408a91b4c6d7359ee555" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 44891 } } peers { permanent_uuid: "a3a2a4e2431646ccb45c9a521a5b5255" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 37545 } } }
I20260928 16:53:12.772779 23365 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a2a4e2431646ccb45c9a521a5b5255 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:53:12.773784 23400 catalog_manager.cc:1347] Generated new cluster ID: f247a534bcef4beeb1d89cc05b620d86
I20260928 16:53:12.773931 23400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:53:12.812103 23400 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:53:12.814268 23400 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:53:12.835282 23400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 061d8cab3f3948218138195770bb08e6: Generated new TSK 0
I20260928 16:53:12.836935 23400 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:53:13.333379 23380 catalog_manager.cc:1259] Loaded cluster ID: f247a534bcef4beeb1d89cc05b620d86
I20260928 16:53:13.333488 23380 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a3a2a4e2431646ccb45c9a521a5b5255: loading cluster ID for follower catalog manager: success
I20260928 16:53:13.336490 23380 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a3a2a4e2431646ccb45c9a521a5b5255: acquiring CA information for follower catalog manager: success
I20260928 16:53:13.338685 23380 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a3a2a4e2431646ccb45c9a521a5b5255: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:53:13.493415 23396 catalog_manager.cc:1259] Loaded cluster ID: f247a534bcef4beeb1d89cc05b620d86
I20260928 16:53:13.493536 23396 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4d521a45079a408a91b4c6d7359ee555: loading cluster ID for follower catalog manager: success
I20260928 16:53:13.496686 23396 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4d521a45079a408a91b4c6d7359ee555: acquiring CA information for follower catalog manager: success
I20260928 16:53:13.498967 23396 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4d521a45079a408a91b4c6d7359ee555: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:53:13.947609 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 23124
I20260928 16:53:13.949199 23186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:14.066265 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 23124
I20260928 16:53:14.088901 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 23198
I20260928 16:53:14.090240 23260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:14.203272 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 23198
I20260928 16:53:14.223711 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 23273
I20260928 16:53:14.225183 23336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:14.339524 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 23273
2026-09-28T16:53:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5078 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:53:14 dist-test-slave-wzcb krb5kdc[23419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:53:14 dist-test-slave-wzcb krb5kdc[23419](info): set up 2 sockets
Sep 28 16:53:14 dist-test-slave-wzcb krb5kdc[23419](info): commencing operation
krb5kdc: starting...
W20260928 16:53:15.410535 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:53:15 dist-test-slave-wzcb krb5kdc[23419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:53:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:53:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:53:15.680166 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:34675
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:35543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:34675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:53:16.050591 23432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:16.050948 23432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:16.051067 23432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:16.058993 23432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:53:16.059139 23432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:16.059198 23432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:16.059281 23432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:53:16.059382 23432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:53:16.070384 23432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:35543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:34675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:34675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:16.072269 23432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:16.073969 23432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:16.088016 23438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:16.087985 23440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:16.088320 23437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:16.090034 23432 server_base.cc:1034] running on GCE node
I20260928 16:53:16.091066 23432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:16.092942 23432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:16.094221 23432 hybrid_clock.cc:648] HybridClock initialized: now 1790614396094145 us; error 67 us; skew 500 ppm
I20260928 16:53:16.094645 23432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:16 dist-test-slave-wzcb krb5kdc[23419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614396, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:16.098304 23432 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:53:16.100409 23432 webserver.cc:460] Webserver started at http://127.9.93.62:38249/ using document root <none> and password file <none>
I20260928 16:53:16.101298 23432 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:16.101445 23432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:53:16.101804 23432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:53:16.105451 23432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "56e30d91a9284009a5630dcc853f123e"
format_stamp: "Formatted at 2026-09-28 16:53:16 on dist-test-slave-wzcb"
server_key: "20cba9567d7fb7d441943c5d8da78c59"
server_key_iv: "9fcc13421d499d9cbe60cbf8850624cb"
server_key_version: "encryptionkey@0"
I20260928 16:53:16.106375 23432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "56e30d91a9284009a5630dcc853f123e"
format_stamp: "Formatted at 2026-09-28 16:53:16 on dist-test-slave-wzcb"
server_key: "20cba9567d7fb7d441943c5d8da78c59"
server_key_iv: "9fcc13421d499d9cbe60cbf8850624cb"
server_key_version: "encryptionkey@0"
I20260928 16:53:16.112993 23432 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 16:53:16.117826 23447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:16.119635 23432 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:53:16.119861 23432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "56e30d91a9284009a5630dcc853f123e"
format_stamp: "Formatted at 2026-09-28 16:53:16 on dist-test-slave-wzcb"
server_key: "20cba9567d7fb7d441943c5d8da78c59"
server_key_iv: "9fcc13421d499d9cbe60cbf8850624cb"
server_key_version: "encryptionkey@0"
I20260928 16:53:16.120113 23432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:16.141414 23432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:16.146341 23432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:16.146665 23432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:16.166440 23432 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:34675
I20260928 16:53:16.166465 23498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:34675 every 8 connection(s)
I20260928 16:53:16.168069 23432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:53:16.172734 23499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:53:16.178503 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 23432
I20260928 16:53:16.178756 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:53:16.179538 9588 external_mini_cluster.cc:1428] Setting key 0ae1837c57559dfe6bbe1677a78da673
I20260928 16:53:16.186534 23499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e30d91a9284009a5630dcc853f123e: Bootstrap starting.
I20260928 16:53:16.191371 23499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56e30d91a9284009a5630dcc853f123e: Neither blocks nor log segments found. Creating new log.
I20260928 16:53:16.192947 23499 log.cc:824] T 00000000000000000000000000000000 P 56e30d91a9284009a5630dcc853f123e: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:53:16 dist-test-slave-wzcb krb5kdc[23419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:53:16.197467 23499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e30d91a9284009a5630dcc853f123e: No bootstrap required, opened a new log
I20260928 16:53:16.207649 23502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:16.181706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37502 (local address 127.9.93.62:34675)
0928 16:53:16.182651 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:16.182665 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:16.182715 (+ 50us) server_negotiation.cc:407] Connection header received
0928 16:53:16.184751 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:16.184784 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:16.185285 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:16.185688 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:16.188025 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:16.189585 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:16.191728 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:16.192635 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:16.192801 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:16.198646 (+ 5845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:16.198656 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:16.198676 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:16.198727 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:16.202126 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:16.203266 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:16.203276 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:16.203284 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:16.203569 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:16.204115 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:16.204119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:16.204121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:16.204853 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:16.205098 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:16.205776 (+ 678us) server_negotiation.cc:299] Negotiation successful
0928 16:53:16.206251 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":414,"threads_started":1}
I20260928 16:53:16.207975 23499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56e30d91a9284009a5630dcc853f123e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e30d91a9284009a5630dcc853f123e" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 34675 } }
I20260928 16:53:16.208629 23499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56e30d91a9284009a5630dcc853f123e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:53:16.208772 23499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56e30d91a9284009a5630dcc853f123e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e30d91a9284009a5630dcc853f123e, State: Initialized, Role: FOLLOWER
I20260928 16:53:16.209581 23499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56e30d91a9284009a5630dcc853f123e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e30d91a9284009a5630dcc853f123e" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 34675 } }
I20260928 16:53:16.209887 23499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56e30d91a9284009a5630dcc853f123e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:53:16.210068 23499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56e30d91a9284009a5630dcc853f123e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:53:16.210265 23499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56e30d91a9284009a5630dcc853f123e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:53:16.212734 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:53:16.213548 23499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56e30d91a9284009a5630dcc853f123e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e30d91a9284009a5630dcc853f123e" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 34675 } }
I20260928 16:53:16.214183 23499 leader_election.cc:302] T 00000000000000000000000000000000 P 56e30d91a9284009a5630dcc853f123e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56e30d91a9284009a5630dcc853f123e; no voters:
I20260928 16:53:16.214926 23499 leader_election.cc:288] T 00000000000000000000000000000000 P 56e30d91a9284009a5630dcc853f123e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:53:16.215095 23504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56e30d91a9284009a5630dcc853f123e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:53:16.216394 23504 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56e30d91a9284009a5630dcc853f123e [term 1 LEADER]: Becoming Leader. State: Replica: 56e30d91a9284009a5630dcc853f123e, State: Running, Role: LEADER
I20260928 16:53:16.217126 23504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56e30d91a9284009a5630dcc853f123e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e30d91a9284009a5630dcc853f123e" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 34675 } }
I20260928 16:53:16.217768 23499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56e30d91a9284009a5630dcc853f123e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:53:16.220086 23505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e30d91a9284009a5630dcc853f123e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56e30d91a9284009a5630dcc853f123e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e30d91a9284009a5630dcc853f123e" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 34675 } } }
I20260928 16:53:16.220436 23506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e30d91a9284009a5630dcc853f123e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56e30d91a9284009a5630dcc853f123e. Latest consensus state: current_term: 1 leader_uuid: "56e30d91a9284009a5630dcc853f123e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e30d91a9284009a5630dcc853f123e" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 34675 } } }
I20260928 16:53:16.220715 23505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e30d91a9284009a5630dcc853f123e [sys.catalog]: This master's current role is: LEADER
I20260928 16:53:16.220715 23506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e30d91a9284009a5630dcc853f123e [sys.catalog]: This master's current role is: LEADER
I20260928 16:53:16.221550 23512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:53:16.228807 23512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:53:16.239233 23512 catalog_manager.cc:1347] Generated new cluster ID: 5e5c10e3e4d54cc9aef2db05c46142e5
I20260928 16:53:16.239405 23512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:53:16.288800 23512 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:53:16.290264 23512 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:53:16.298604 23512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56e30d91a9284009a5630dcc853f123e: Generated new TSK 0
I20260928 16:53:16.299715 23512 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:53:16.326900 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 23432
I20260928 16:53:16.328434 23494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:16.435425 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 23432
2026-09-28T16:53:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2093 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:53:16 dist-test-slave-wzcb krb5kdc[23527](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:53:16 dist-test-slave-wzcb krb5kdc[23527](info): set up 2 sockets
Sep 28 16:53:16 dist-test-slave-wzcb krb5kdc[23527](info): commencing operation
krb5kdc: starting...
W20260928 16:53:17.470271 9588 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:53:17 dist-test-slave-wzcb krb5kdc[23527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:53:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:53:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:53:17.735859 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:42273
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:42273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:53:18.104475 23540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:18.104846 23540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:18.104979 23540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:18.112824 23540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:53:18.112968 23540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:18.113018 23540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:18.113103 23540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:53:18.113168 23540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:53:18.123946 23540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:42273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:42273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:18.125552 23540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:18.127102 23540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:18.140259 23545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:18.140204 23548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:18.140210 23546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:18.141239 23540 server_base.cc:1034] running on GCE node
I20260928 16:53:18.142220 23540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:18.143996 23540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:18.145251 23540 hybrid_clock.cc:648] HybridClock initialized: now 1790614398145185 us; error 51 us; skew 500 ppm
I20260928 16:53:18.145697 23540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:18 dist-test-slave-wzcb krb5kdc[23527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614398, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:18.149164 23540 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:53:18.150960 23540 webserver.cc:460] Webserver started at http://127.9.93.62:43217/ using document root <none> and password file <none>
I20260928 16:53:18.151819 23540 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:18.151948 23540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:53:18.152263 23540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:53:18.155778 23540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "68bb897b18f246eea07a972893292f25"
format_stamp: "Formatted at 2026-09-28 16:53:18 on dist-test-slave-wzcb"
server_key: "695157bd7fa2b586e02aa30321ed74c2"
server_key_iv: "502338b6206297f99a75949202791092"
server_key_version: "encryptionkey@0"
I20260928 16:53:18.156718 23540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "68bb897b18f246eea07a972893292f25"
format_stamp: "Formatted at 2026-09-28 16:53:18 on dist-test-slave-wzcb"
server_key: "695157bd7fa2b586e02aa30321ed74c2"
server_key_iv: "502338b6206297f99a75949202791092"
server_key_version: "encryptionkey@0"
I20260928 16:53:18.163198 23540 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 16:53:18.167945 23555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:18.170181 23540 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:53:18.170467 23540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "68bb897b18f246eea07a972893292f25"
format_stamp: "Formatted at 2026-09-28 16:53:18 on dist-test-slave-wzcb"
server_key: "695157bd7fa2b586e02aa30321ed74c2"
server_key_iv: "502338b6206297f99a75949202791092"
server_key_version: "encryptionkey@0"
I20260928 16:53:18.170723 23540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:18.196775 23540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:18.201961 23540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:18.202284 23540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:18.222692 23540 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:42273
I20260928 16:53:18.222714 23606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:42273 every 8 connection(s)
I20260928 16:53:18.224270 23540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:53:18.228775 23607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:53:18.234027 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 23540
I20260928 16:53:18.234273 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:53:18.235163 9588 external_mini_cluster.cc:1428] Setting key 437b7d9755889facca0089290bc75ee8
I20260928 16:53:18.242533 23607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25: Bootstrap starting.
I20260928 16:53:18.246845 23607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25: Neither blocks nor log segments found. Creating new log.
I20260928 16:53:18.248310 23607 log.cc:824] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:53:18 dist-test-slave-wzcb krb5kdc[23527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:53:18.252573 23607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25: No bootstrap required, opened a new log
I20260928 16:53:18.261952 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:18.237717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45906 (local address 127.9.93.62:42273)
0928 16:53:18.238575 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:18.238589 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:18.238641 (+ 52us) server_negotiation.cc:407] Connection header received
0928 16:53:18.240711 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:18.240749 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:18.241238 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:18.241625 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:18.243620 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:18.245227 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:18.247231 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:18.248093 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:18.248256 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:18.253440 (+ 5184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:18.253450 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:18.253468 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:18.253533 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:18.256985 (+ 3452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:18.257975 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:18.257982 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:18.257990 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:18.258208 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:18.258660 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:18.258663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:18.258665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:18.259445 (+ 780us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:18.259719 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:18.260229 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 16:53:18.260721 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":380,"threads_started":1}
I20260928 16:53:18.262285 23607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bb897b18f246eea07a972893292f25" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42273 } }
I20260928 16:53:18.262841 23607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:53:18.262971 23607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68bb897b18f246eea07a972893292f25, State: Initialized, Role: FOLLOWER
I20260928 16:53:18.263736 23607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bb897b18f246eea07a972893292f25" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42273 } }
I20260928 16:53:18.263988 23607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:53:18.264132 23607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:53:18.264312 23607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:53:18.266960 23607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bb897b18f246eea07a972893292f25" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42273 } }
I20260928 16:53:18.267664 23607 leader_election.cc:302] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68bb897b18f246eea07a972893292f25; no voters:
I20260928 16:53:18.268363 23607 leader_election.cc:288] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:53:18.268507 23612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:53:18.269155 23612 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [term 1 LEADER]: Becoming Leader. State: Replica: 68bb897b18f246eea07a972893292f25, State: Running, Role: LEADER
I20260928 16:53:18.269774 23612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bb897b18f246eea07a972893292f25" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42273 } }
I20260928 16:53:18.270550 23607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:53:18.272951 23613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68bb897b18f246eea07a972893292f25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bb897b18f246eea07a972893292f25" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42273 } } }
I20260928 16:53:18.273068 23614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68bb897b18f246eea07a972893292f25. Latest consensus state: current_term: 1 leader_uuid: "68bb897b18f246eea07a972893292f25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bb897b18f246eea07a972893292f25" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42273 } } }
I20260928 16:53:18.273376 23613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [sys.catalog]: This master's current role is: LEADER
I20260928 16:53:18.273456 23614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [sys.catalog]: This master's current role is: LEADER
I20260928 16:53:18.276620 23620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:53:18.281713 23620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:53:18.292531 23620 catalog_manager.cc:1347] Generated new cluster ID: f3a1f2102e814db08c4f8360a8290d0b
I20260928 16:53:18.292676 23620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:53:18.314668 23620 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:53:18.316885 23620 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:53:18.325438 23620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25: Generated new TSK 0
I20260928 16:53:18.326797 23620 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
I20260928 16:53:18.391417 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:42273
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:53:18.782956 23633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:18.783326 23633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:18.783457 23633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:18.791682 23633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:18.791836 23633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:18.791961 23633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:53:18.803098 23633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:42273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:18.804685 23633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:18.806275 23633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:18.821720 23639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:18.821789 23638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:18.821753 23641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:18.822841 23633 server_base.cc:1034] running on GCE node
I20260928 16:53:18.823647 23633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:18.824930 23633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:18.826197 23633 hybrid_clock.cc:648] HybridClock initialized: now 1790614398826141 us; error 62 us; skew 500 ppm
I20260928 16:53:18.826586 23633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:18 dist-test-slave-wzcb krb5kdc[23527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614398, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:18.830358 23633 init.cc:377] Logged in from keytab as kudu/127.9.93.1@KRBTEST.COM (short username kudu)
I20260928 16:53:18.832271 23633 webserver.cc:460] Webserver started at http://127.9.93.1:33071/ using document root <none> and password file <none>
I20260928 16:53:18.833132 23633 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:18.833252 23633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:53:18.833614 23633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:53:18.837209 23633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "6d09b4c117584c7b8ac25d3615f00d4e"
format_stamp: "Formatted at 2026-09-28 16:53:18 on dist-test-slave-wzcb"
server_key: "f5281552e90275b15050134742f985bd"
server_key_iv: "a0f9355b7f632842857dd5a02bb1b869"
server_key_version: "encryptionkey@0"
I20260928 16:53:18.838163 23633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "6d09b4c117584c7b8ac25d3615f00d4e"
format_stamp: "Formatted at 2026-09-28 16:53:18 on dist-test-slave-wzcb"
server_key: "f5281552e90275b15050134742f985bd"
server_key_iv: "a0f9355b7f632842857dd5a02bb1b869"
server_key_version: "encryptionkey@0"
I20260928 16:53:18.844595 23633 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 16:53:18.848897 23648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:18.850585 23633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260928 16:53:18.850782 23633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "6d09b4c117584c7b8ac25d3615f00d4e"
format_stamp: "Formatted at 2026-09-28 16:53:18 on dist-test-slave-wzcb"
server_key: "f5281552e90275b15050134742f985bd"
server_key_iv: "a0f9355b7f632842857dd5a02bb1b869"
server_key_version: "encryptionkey@0"
I20260928 16:53:18.851018 23633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:18.881606 23633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:18.886245 23633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:18.886559 23633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:18.887763 23633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:53:18.889401 23633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:53:18.889484 23633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:18.889598 23633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:53:18.889667 23633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:18.922725 23633 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:38771
I20260928 16:53:18.922794 23760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:38771 every 8 connection(s)
I20260928 16:53:18.924175 23633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:53:18.931421 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 23633
I20260928 16:53:18.931670 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:53:18.932340 9588 external_mini_cluster.cc:1428] Setting key df023f78c3285f9b7a7a396d68d3af97
Sep 28 16:53:18 dist-test-slave-wzcb krb5kdc[23527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614398, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:53:18.947184 23763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:18.926956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:37129 (local address 127.9.93.62:42273)
0928 16:53:18.927510 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:18.927515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:18.928689 (+ 1174us) server_negotiation.cc:407] Connection header received
0928 16:53:18.930753 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:18.930759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:18.930875 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:18.931010 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:18.932965 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:18.933705 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:18.935719 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:18.936161 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:18.936279 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:18.940306 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:18.940314 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:18.940319 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:18.940364 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:18.943448 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:18.944479 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:18.944484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:18.944487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:18.944641 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:18.945231 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:18.945236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:18.945238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:18.945560 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:18.945708 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:18.946655 (+ 947us) server_negotiation.cc:299] Negotiation successful
0928 16:53:18.946871 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":250,"threads_started":1}
I20260928 16:53:18.948803 23764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:18.927673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.1:37129)
0928 16:53:18.928497 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:18.928550 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:18.930406 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:18.931288 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:18.931329 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:18.932029 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:53:18.932750 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:18.932773 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:18.933894 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:18.933900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:18.935553 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:18.935567 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:18.936412 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:18.936421 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:18.936681 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:18.937371 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:53:18.937395 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:53:18.939979 (+ 2584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:53:18.943626 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:18.943635 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:18.943651 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:18.944294 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:18.944759 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:18.944764 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:18.944767 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:18.945068 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:18.945708 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:53:18.945717 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:53:18.946224 (+ 507us) client_negotiation.cc:769] Sending connection context
0928 16:53:18.946539 (+ 315us) client_negotiation.cc:240] Negotiation successful
0928 16:53:18.947169 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":309,"threads_started":1}
I20260928 16:53:18.950642 23761 heartbeater.cc:347] Connected to a master server at 127.9.93.62:42273
I20260928 16:53:18.951037 23761 heartbeater.cc:464] Registering TS with master...
I20260928 16:53:18.951989 23761 heartbeater.cc:511] Master 127.9.93.62:42273 requested a full tablet report, sending...
I20260928 16:53:18.954497 23572 ts_manager.cc:194] Registered new tserver with Master: 6d09b4c117584c7b8ac25d3615f00d4e (127.9.93.1:38771)
I20260928 16:53:18.956279 23572 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.1@KRBTEST.COM'} at 127.9.93.1:37129
WARNING: no policy specified for kudu/127.9.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
I20260928 16:53:18.992877 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:42273
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:53:19.349198 23767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:19.349555 23767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:19.349709 23767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:19.357710 23767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:19.357858 23767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:19.358016 23767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:53:19.368554 23767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:42273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:19.370051 23767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:19.371685 23767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:19.385962 23772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:19.385929 23773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:19.385921 23775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:19.386837 23767 server_base.cc:1034] running on GCE node
I20260928 16:53:19.387579 23767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:19.388625 23767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:19.389824 23767 hybrid_clock.cc:648] HybridClock initialized: now 1790614399389756 us; error 60 us; skew 500 ppm
I20260928 16:53:19.390156 23767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:19 dist-test-slave-wzcb krb5kdc[23527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614399, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:19.393417 23767 init.cc:377] Logged in from keytab as kudu/127.9.93.2@KRBTEST.COM (short username kudu)
I20260928 16:53:19.394997 23767 webserver.cc:460] Webserver started at http://127.9.93.2:36589/ using document root <none> and password file <none>
I20260928 16:53:19.395818 23767 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:19.395936 23767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:53:19.396253 23767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:53:19.399480 23767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "d0592c06e71c4c528911f2095888a885"
format_stamp: "Formatted at 2026-09-28 16:53:19 on dist-test-slave-wzcb"
server_key: "4aef89440d6683b3042ff3e0a2dbdc81"
server_key_iv: "eb327cd159e69a7d23bde1adc99c30ed"
server_key_version: "encryptionkey@0"
I20260928 16:53:19.400300 23767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "d0592c06e71c4c528911f2095888a885"
format_stamp: "Formatted at 2026-09-28 16:53:19 on dist-test-slave-wzcb"
server_key: "4aef89440d6683b3042ff3e0a2dbdc81"
server_key_iv: "eb327cd159e69a7d23bde1adc99c30ed"
server_key_version: "encryptionkey@0"
I20260928 16:53:19.406260 23767 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260928 16:53:19.410746 23782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:19.412318 23767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:53:19.412518 23767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "d0592c06e71c4c528911f2095888a885"
format_stamp: "Formatted at 2026-09-28 16:53:19 on dist-test-slave-wzcb"
server_key: "4aef89440d6683b3042ff3e0a2dbdc81"
server_key_iv: "eb327cd159e69a7d23bde1adc99c30ed"
server_key_version: "encryptionkey@0"
I20260928 16:53:19.412721 23767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:19.426035 23767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:19.436074 23767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:19.436393 23767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:19.437461 23767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:53:19.439031 23767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:53:19.439117 23767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:19.439245 23767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:53:19.439324 23767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:19.470206 23767 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:41983
I20260928 16:53:19.470234 23894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:41983 every 8 connection(s)
I20260928 16:53:19.471663 23767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:53:19.479718 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 23767
I20260928 16:53:19.479904 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:53:19.480386 9588 external_mini_cluster.cc:1428] Setting key 60c5a36e274ca9992e05d9ca88f1f6ab
Sep 28 16:53:19 dist-test-slave-wzcb krb5kdc[23527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614399, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:53:19.492913 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:19.474256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:37577 (local address 127.9.93.62:42273)
0928 16:53:19.474787 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:19.474794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:19.475986 (+ 1192us) server_negotiation.cc:407] Connection header received
0928 16:53:19.477902 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:19.477907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:19.477986 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:19.478093 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:19.479768 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:19.480588 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:19.482371 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:19.482974 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:19.483122 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:19.486899 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:19.486905 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:19.486908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:19.486944 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:19.489589 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:19.490461 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:19.490465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:19.490467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:19.490595 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:19.491033 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:19.491036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:19.491038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:19.491294 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:19.491461 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:19.492331 (+ 870us) server_negotiation.cc:299] Negotiation successful
0928 16:53:19.492540 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":268,"threads_started":1}
I20260928 16:53:19.494067 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:19.474886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.2:37577)
0928 16:53:19.475810 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:19.475865 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:19.477638 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:19.478309 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:19.478319 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:19.478871 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:53:19.479569 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:19.479591 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:19.480741 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:19.480745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:19.482227 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:19.482244 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:19.483225 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:19.483231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:19.483510 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:19.484172 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:53:19.484196 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:53:19.486631 (+ 2435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:53:19.489735 (+ 3104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:19.489744 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:19.489763 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:19.490341 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:19.490692 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:19.490696 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:19.490699 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:19.490933 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:19.491462 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:53:19.491471 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:53:19.491947 (+ 476us) client_negotiation.cc:769] Sending connection context
0928 16:53:19.492262 (+ 315us) client_negotiation.cc:240] Negotiation successful
0928 16:53:19.492812 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":343,"threads_started":1}
I20260928 16:53:19.495577 23895 heartbeater.cc:347] Connected to a master server at 127.9.93.62:42273
I20260928 16:53:19.495940 23895 heartbeater.cc:464] Registering TS with master...
I20260928 16:53:19.496865 23895 heartbeater.cc:511] Master 127.9.93.62:42273 requested a full tablet report, sending...
I20260928 16:53:19.498672 23572 ts_manager.cc:194] Registered new tserver with Master: d0592c06e71c4c528911f2095888a885 (127.9.93.2:41983)
I20260928 16:53:19.499588 23572 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.2@KRBTEST.COM'} at 127.9.93.2:37577
WARNING: no policy specified for kudu/127.9.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
I20260928 16:53:19.539368 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:42273
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:53:19.899996 23901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:19.900396 23901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:19.900559 23901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:19.908682 23901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:19.908849 23901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:19.909016 23901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:53:19.919682 23901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:42273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:19.921197 23901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:19.922766 23901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:19.937184 23907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:19.937081 23906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:19.937158 23909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:19.938177 23901 server_base.cc:1034] running on GCE node
I20260928 16:53:19.938977 23901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:19.940145 23901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:19.941390 23901 hybrid_clock.cc:648] HybridClock initialized: now 1790614399941312 us; error 60 us; skew 500 ppm
I20260928 16:53:19.941802 23901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:19 dist-test-slave-wzcb krb5kdc[23527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614399, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:19.945493 23901 init.cc:377] Logged in from keytab as kudu/127.9.93.3@KRBTEST.COM (short username kudu)
I20260928 16:53:19.947175 23901 webserver.cc:460] Webserver started at http://127.9.93.3:36985/ using document root <none> and password file <none>
I20260928 16:53:19.948057 23901 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:19.948190 23901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:53:19.948515 23901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:53:19.951920 23901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "37a490e2b2c44e5d8d7b60a4af15312b"
format_stamp: "Formatted at 2026-09-28 16:53:19 on dist-test-slave-wzcb"
server_key: "2c30f5ea8ad729eb78b56b18f154d733"
server_key_iv: "cfd49a00e49e987d9a23e9b4e97e37c2"
server_key_version: "encryptionkey@0"
I20260928 16:53:19.952780 23901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "37a490e2b2c44e5d8d7b60a4af15312b"
format_stamp: "Formatted at 2026-09-28 16:53:19 on dist-test-slave-wzcb"
server_key: "2c30f5ea8ad729eb78b56b18f154d733"
server_key_iv: "cfd49a00e49e987d9a23e9b4e97e37c2"
server_key_version: "encryptionkey@0"
I20260928 16:53:19.959007 23901 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 16:53:19.959954 23761 heartbeater.cc:503] Master 127.9.93.62:42273 was elected leader, sending a full tablet report...
I20260928 16:53:19.963241 23916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:19.964947 23901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:53:19.965147 23901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "37a490e2b2c44e5d8d7b60a4af15312b"
format_stamp: "Formatted at 2026-09-28 16:53:19 on dist-test-slave-wzcb"
server_key: "2c30f5ea8ad729eb78b56b18f154d733"
server_key_iv: "cfd49a00e49e987d9a23e9b4e97e37c2"
server_key_version: "encryptionkey@0"
I20260928 16:53:19.965371 23901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:19.989910 23901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:19.994876 23901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:19.995179 23901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:19.996320 23901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:53:19.997887 23901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:53:19.997973 23901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:19.998103 23901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:53:19.998164 23901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:20.030979 23901 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:34841
I20260928 16:53:20.031042 24028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:34841 every 8 connection(s)
I20260928 16:53:20.032478 23901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:53:20.035820 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 23901
I20260928 16:53:20.036062 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:53:20.036706 9588 external_mini_cluster.cc:1428] Setting key 061adfc0a0fd03c1529f4132db7efd19
Sep 28 16:53:20 dist-test-slave-wzcb krb5kdc[23527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614399, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:53:20.055195 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:20.035271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:35591 (local address 127.9.93.62:42273)
0928 16:53:20.035839 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:20.035844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:20.037203 (+ 1359us) server_negotiation.cc:407] Connection header received
0928 16:53:20.039746 (+ 2543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:20.039751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:20.039880 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:20.040014 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:20.041870 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.042674 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.044396 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.044821 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.044946 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:20.048646 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:20.048652 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:20.048656 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:20.048696 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:20.051506 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:20.052449 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:20.052460 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:20.052463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:20.052660 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:20.053190 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:20.053194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:20.053196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:20.053499 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:20.053646 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:20.054450 (+ 804us) server_negotiation.cc:299] Negotiation successful
0928 16:53:20.054723 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":283,"threads_started":1}
I20260928 16:53:20.056373 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:20.036058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.3:35591)
0928 16:53:20.037018 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:20.037069 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:20.039372 (+ 2303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:20.040341 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:20.040352 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:20.040959 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:53:20.041682 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:20.041702 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.042807 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.042811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:20.044241 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:20.044254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.045042 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.045047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:20.045276 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:20.045924 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:53:20.045948 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:53:20.048391 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:53:20.051654 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:20.051663 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:20.051679 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:20.052306 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:20.052780 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:20.052785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:20.052788 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:20.053065 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:20.053639 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:53:20.053648 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:53:20.054040 (+ 392us) client_negotiation.cc:769] Sending connection context
0928 16:53:20.054303 (+ 263us) client_negotiation.cc:240] Negotiation successful
0928 16:53:20.054980 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":398,"threads_started":1}
I20260928 16:53:20.058295 24029 heartbeater.cc:347] Connected to a master server at 127.9.93.62:42273
I20260928 16:53:20.058660 24029 heartbeater.cc:464] Registering TS with master...
I20260928 16:53:20.059588 24029 heartbeater.cc:511] Master 127.9.93.62:42273 requested a full tablet report, sending...
I20260928 16:53:20.061360 23572 ts_manager.cc:194] Registered new tserver with Master: 37a490e2b2c44e5d8d7b60a4af15312b (127.9.93.3:34841)
I20260928 16:53:20.062234 23572 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.3@KRBTEST.COM'} at 127.9.93.3:35591
I20260928 16:53:20.065925 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:53:20.085114 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:20.070838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45908 (local address 127.9.93.62:42273)
0928 16:53:20.071063 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:20.071072 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:20.071642 (+ 570us) server_negotiation.cc:407] Connection header received
0928 16:53:20.071850 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:20.071854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:20.071946 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:20.072038 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:20.073741 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.074683 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.076390 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.076840 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.076948 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:20.079669 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:20.079674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:20.079678 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:20.079722 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:20.082439 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:20.083174 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:20.083177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:20.083179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:20.083352 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:20.083707 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:20.083710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:20.083712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:20.084077 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:20.084270 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:20.084603 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 16:53:20.084804 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 16:53:20.089687 23572 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:53:20.097064 23572 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:53:20.125833 24043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:20.111180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:38771 (local address 127.0.0.1:39120)
0928 16:53:20.112958 (+ 1778us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:20.112976 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:20.113084 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:20.114986 (+ 1902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:20.114995 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:20.115036 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:53:20.117564 (+ 2528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:20.117578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.119711 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.119717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:20.122254 (+ 2537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:20.122271 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.124714 (+ 2443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.124721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:20.124965 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:20.125040 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 16:53:20.125177 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 16:53:20.125418 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1649,"thread_start_us":1505,"threads_started":1}
I20260928 16:53:20.127084 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:20.112291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39120 (local address 127.9.93.1:38771)
0928 16:53:20.113871 (+ 1580us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:20.113881 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:20.113915 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:53:20.114036 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:20.114046 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:20.114402 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:20.114632 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:53:20.117828 (+ 3196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.119522 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.122463 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.124121 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.124269 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:20.124569 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:20.126492 (+ 1923us) server_negotiation.cc:299] Negotiation successful
0928 16:53:20.126706 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1394,"thread_start_us":1227,"threads_started":1}
I20260928 16:53:20.129032 24042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:20.111157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:34841 (local address 127.0.0.1:47716)
0928 16:53:20.112070 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:20.112131 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:20.112341 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:20.118404 (+ 6063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:20.118410 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:20.118441 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:53:20.118840 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:20.118853 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.122421 (+ 3568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.122427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:20.126052 (+ 3625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:20.126069 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.128089 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.128097 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:20.128297 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:20.128332 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:53:20.128427 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 16:53:20.128601 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":1,"thread_start_us":425,"threads_started":1}
I20260928 16:53:20.129956 24044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:20.111669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47716 (local address 127.9.93.3:34841)
0928 16:53:20.117324 (+ 5655us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:20.117333 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:20.117387 (+ 54us) server_negotiation.cc:407] Connection header received
0928 16:53:20.117499 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:20.117509 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:20.117843 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:20.118073 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:53:20.119056 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.122216 (+ 3160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.126277 (+ 4061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.127861 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.128004 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:20.128257 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:20.129372 (+ 1115us) server_negotiation.cc:299] Negotiation successful
0928 16:53:20.129592 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5483,"thread_start_us":5327,"threads_started":1}
I20260928 16:53:20.131942 23696 tablet_service.cc:1467] Processing CreateTablet for tablet bd023ad546114d818753b5509d03bdcf (DEFAULT_TABLE table=test-table [id=443c226734224cdd86a0974f82dbe77e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:53:20.133816 23696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd023ad546114d818753b5509d03bdcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:53:20.134677 24043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:20.113088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:41983 (local address 127.0.0.1:41012)
0928 16:53:20.126163 (+ 13075us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:20.126183 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:20.126297 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:20.128462 (+ 2165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:20.128468 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:20.128499 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:53:20.128657 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:20.128669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.130619 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.130625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:20.132363 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:20.132378 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.133767 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.133773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:20.134089 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:20.134117 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 16:53:20.134202 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 16:53:20.134381 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12920,"thread_start_us":13465,"threads_started":1}
I20260928 16:53:20.134821 24047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:20.113610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41012 (local address 127.9.93.2:41983)
0928 16:53:20.127590 (+ 13980us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:20.127600 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:20.127634 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:53:20.127728 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:20.127734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:20.127965 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:20.128214 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:53:20.128852 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.130421 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.132547 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.133583 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.133711 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:20.133921 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:20.134286 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 16:53:20.134459 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13755,"spinlock_wait_cycles":29056,"thread_start_us":13725,"threads_started":1}
I20260928 16:53:20.134506 23964 tablet_service.cc:1467] Processing CreateTablet for tablet bd023ad546114d818753b5509d03bdcf (DEFAULT_TABLE table=test-table [id=443c226734224cdd86a0974f82dbe77e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:53:20.136269 23964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd023ad546114d818753b5509d03bdcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:53:20.138753 23830 tablet_service.cc:1467] Processing CreateTablet for tablet bd023ad546114d818753b5509d03bdcf (DEFAULT_TABLE table=test-table [id=443c226734224cdd86a0974f82dbe77e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:53:20.140192 23830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd023ad546114d818753b5509d03bdcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:53:20.150627 24048 tablet_bootstrap.cc:492] T bd023ad546114d818753b5509d03bdcf P 6d09b4c117584c7b8ac25d3615f00d4e: Bootstrap starting.
I20260928 16:53:20.153033 24049 tablet_bootstrap.cc:492] T bd023ad546114d818753b5509d03bdcf P d0592c06e71c4c528911f2095888a885: Bootstrap starting.
I20260928 16:53:20.154551 24050 tablet_bootstrap.cc:492] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b: Bootstrap starting.
I20260928 16:53:20.156241 24048 tablet_bootstrap.cc:654] T bd023ad546114d818753b5509d03bdcf P 6d09b4c117584c7b8ac25d3615f00d4e: Neither blocks nor log segments found. Creating new log.
I20260928 16:53:20.157039 24049 tablet_bootstrap.cc:654] T bd023ad546114d818753b5509d03bdcf P d0592c06e71c4c528911f2095888a885: Neither blocks nor log segments found. Creating new log.
I20260928 16:53:20.157864 24048 log.cc:824] T bd023ad546114d818753b5509d03bdcf P 6d09b4c117584c7b8ac25d3615f00d4e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:53:20.158141 24049 log.cc:824] T bd023ad546114d818753b5509d03bdcf P d0592c06e71c4c528911f2095888a885: Log is configured to *not* fsync() on all Append() calls
I20260928 16:53:20.158768 24050 tablet_bootstrap.cc:654] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b: Neither blocks nor log segments found. Creating new log.
I20260928 16:53:20.160296 24050 log.cc:824] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b: Log is configured to *not* fsync() on all Append() calls
I20260928 16:53:20.161715 24048 tablet_bootstrap.cc:492] T bd023ad546114d818753b5509d03bdcf P 6d09b4c117584c7b8ac25d3615f00d4e: No bootstrap required, opened a new log
I20260928 16:53:20.162047 24048 ts_tablet_manager.cc:1397] T bd023ad546114d818753b5509d03bdcf P 6d09b4c117584c7b8ac25d3615f00d4e: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260928 16:53:20.162657 24049 tablet_bootstrap.cc:492] T bd023ad546114d818753b5509d03bdcf P d0592c06e71c4c528911f2095888a885: No bootstrap required, opened a new log
I20260928 16:53:20.162935 24049 ts_tablet_manager.cc:1397] T bd023ad546114d818753b5509d03bdcf P d0592c06e71c4c528911f2095888a885: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 16:53:20.163750 24050 tablet_bootstrap.cc:492] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b: No bootstrap required, opened a new log
I20260928 16:53:20.164036 24050 ts_tablet_manager.cc:1397] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260928 16:53:20.171548 24049 raft_consensus.cc:348] T bd023ad546114d818753b5509d03bdcf P d0592c06e71c4c528911f2095888a885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34841 } } peers { permanent_uuid: "6d09b4c117584c7b8ac25d3615f00d4e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38771 } } peers { permanent_uuid: "d0592c06e71c4c528911f2095888a885" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41983 } }
I20260928 16:53:20.172089 24049 raft_consensus.cc:374] T bd023ad546114d818753b5509d03bdcf P d0592c06e71c4c528911f2095888a885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:53:20.172212 24049 raft_consensus.cc:729] T bd023ad546114d818753b5509d03bdcf P d0592c06e71c4c528911f2095888a885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0592c06e71c4c528911f2095888a885, State: Initialized, Role: FOLLOWER
I20260928 16:53:20.172936 24049 consensus_queue.cc:260] T bd023ad546114d818753b5509d03bdcf P d0592c06e71c4c528911f2095888a885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34841 } } peers { permanent_uuid: "6d09b4c117584c7b8ac25d3615f00d4e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38771 } } peers { permanent_uuid: "d0592c06e71c4c528911f2095888a885" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41983 } }
I20260928 16:53:20.173718 24048 raft_consensus.cc:348] T bd023ad546114d818753b5509d03bdcf P 6d09b4c117584c7b8ac25d3615f00d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34841 } } peers { permanent_uuid: "6d09b4c117584c7b8ac25d3615f00d4e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38771 } } peers { permanent_uuid: "d0592c06e71c4c528911f2095888a885" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41983 } }
I20260928 16:53:20.174211 23895 heartbeater.cc:503] Master 127.9.93.62:42273 was elected leader, sending a full tablet report...
I20260928 16:53:20.174249 24048 raft_consensus.cc:374] T bd023ad546114d818753b5509d03bdcf P 6d09b4c117584c7b8ac25d3615f00d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:53:20.174360 24048 raft_consensus.cc:729] T bd023ad546114d818753b5509d03bdcf P 6d09b4c117584c7b8ac25d3615f00d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d09b4c117584c7b8ac25d3615f00d4e, State: Initialized, Role: FOLLOWER
I20260928 16:53:20.175169 24048 consensus_queue.cc:260] T bd023ad546114d818753b5509d03bdcf P 6d09b4c117584c7b8ac25d3615f00d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34841 } } peers { permanent_uuid: "6d09b4c117584c7b8ac25d3615f00d4e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38771 } } peers { permanent_uuid: "d0592c06e71c4c528911f2095888a885" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41983 } }
I20260928 16:53:20.175172 24050 raft_consensus.cc:348] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34841 } } peers { permanent_uuid: "6d09b4c117584c7b8ac25d3615f00d4e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38771 } } peers { permanent_uuid: "d0592c06e71c4c528911f2095888a885" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41983 } }
I20260928 16:53:20.175696 24050 raft_consensus.cc:374] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:53:20.175806 24050 raft_consensus.cc:729] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a490e2b2c44e5d8d7b60a4af15312b, State: Initialized, Role: FOLLOWER
I20260928 16:53:20.176738 24049 ts_tablet_manager.cc:1428] T bd023ad546114d818753b5509d03bdcf P d0592c06e71c4c528911f2095888a885: Time spent starting tablet: real 0.014s user 0.002s sys 0.010s
I20260928 16:53:20.176707 24050 consensus_queue.cc:260] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34841 } } peers { permanent_uuid: "6d09b4c117584c7b8ac25d3615f00d4e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38771 } } peers { permanent_uuid: "d0592c06e71c4c528911f2095888a885" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41983 } }
I20260928 16:53:20.178031 24029 heartbeater.cc:503] Master 127.9.93.62:42273 was elected leader, sending a full tablet report...
I20260928 16:53:20.178869 24050 ts_tablet_manager.cc:1428] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260928 16:53:20.181216 24048 ts_tablet_manager.cc:1428] T bd023ad546114d818753b5509d03bdcf P 6d09b4c117584c7b8ac25d3615f00d4e: Time spent starting tablet: real 0.019s user 0.015s sys 0.005s
I20260928 16:53:20.186766 24056 raft_consensus.cc:482] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:53:20.187096 24056 raft_consensus.cc:504] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34841 } } peers { permanent_uuid: "6d09b4c117584c7b8ac25d3615f00d4e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38771 } } peers { permanent_uuid: "d0592c06e71c4c528911f2095888a885" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41983 } }
I20260928 16:53:20.189031 24056 leader_election.cc:288] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d09b4c117584c7b8ac25d3615f00d4e (127.9.93.1:38771), d0592c06e71c4c528911f2095888a885 (127.9.93.2:41983)
I20260928 16:53:20.194644 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:20.189301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:38771 (local address 127.9.93.3:42243)
0928 16:53:20.189527 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:20.189551 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:20.189688 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:20.190211 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:20.190215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:20.190244 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:53:20.190407 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:20.190418 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.191467 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.191471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:20.192767 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:20.192780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.193901 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.193907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:20.194094 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:20.194120 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:53:20.194204 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 16:53:20.194354 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 16:53:20.194653 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:20.189413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:42243 (local address 127.9.93.1:38771)
0928 16:53:20.189608 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:20.189614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:20.189636 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:53:20.189825 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:20.189830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:20.189908 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:20.189997 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:53:20.190559 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.191325 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.192934 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.193737 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.193866 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:20.193977 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:20.194249 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 16:53:20.194390 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:53:20.195628 24047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:20.189947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:39645 (local address 127.9.93.2:41983)
0928 16:53:20.190149 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:20.190155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:20.190528 (+ 373us) server_negotiation.cc:407] Connection header received
0928 16:53:20.190690 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:20.190696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:20.190762 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:20.190834 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:53:20.191345 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.192066 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.193735 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.194579 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.194703 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:20.194799 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:20.195186 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 16:53:20.195371 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 16:53:20.195984 24057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:20.189758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:41983 (local address 127.9.93.3:39645)
0928 16:53:20.190425 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:20.190440 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:20.190563 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:20.190860 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:20.190864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:20.190888 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:53:20.191182 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:20.191191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.192216 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.192220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:20.193575 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:20.193588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.194790 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.194796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:20.194968 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:20.195000 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 16:53:20.195129 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 16:53:20.195267 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":333,"threads_started":1}
I20260928 16:53:20.195981 23716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd023ad546114d818753b5509d03bdcf" candidate_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d09b4c117584c7b8ac25d3615f00d4e" is_pre_election: true
I20260928 16:53:20.196758 23716 raft_consensus.cc:2454] T bd023ad546114d818753b5509d03bdcf P 6d09b4c117584c7b8ac25d3615f00d4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37a490e2b2c44e5d8d7b60a4af15312b in term 0.
I20260928 16:53:20.197129 23850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd023ad546114d818753b5509d03bdcf" candidate_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0592c06e71c4c528911f2095888a885" is_pre_election: true
I20260928 16:53:20.197508 23918 leader_election.cc:302] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37a490e2b2c44e5d8d7b60a4af15312b, 6d09b4c117584c7b8ac25d3615f00d4e; no voters:
I20260928 16:53:20.197692 23850 raft_consensus.cc:2454] T bd023ad546114d818753b5509d03bdcf P d0592c06e71c4c528911f2095888a885 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37a490e2b2c44e5d8d7b60a4af15312b in term 0.
I20260928 16:53:20.198007 24056 raft_consensus.cc:2781] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:53:20.198155 24056 raft_consensus.cc:482] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:53:20.198266 24056 raft_consensus.cc:3037] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:53:20.200788 24056 raft_consensus.cc:504] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34841 } } peers { permanent_uuid: "6d09b4c117584c7b8ac25d3615f00d4e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38771 } } peers { permanent_uuid: "d0592c06e71c4c528911f2095888a885" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41983 } }
I20260928 16:53:20.201416 24056 leader_election.cc:288] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b [CANDIDATE]: Term 1 election: Requested vote from peers 6d09b4c117584c7b8ac25d3615f00d4e (127.9.93.1:38771), d0592c06e71c4c528911f2095888a885 (127.9.93.2:41983)
I20260928 16:53:20.202067 23716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd023ad546114d818753b5509d03bdcf" candidate_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d09b4c117584c7b8ac25d3615f00d4e"
I20260928 16:53:20.202083 23850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd023ad546114d818753b5509d03bdcf" candidate_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0592c06e71c4c528911f2095888a885"
I20260928 16:53:20.202286 23716 raft_consensus.cc:3037] T bd023ad546114d818753b5509d03bdcf P 6d09b4c117584c7b8ac25d3615f00d4e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:53:20.202288 23850 raft_consensus.cc:3037] T bd023ad546114d818753b5509d03bdcf P d0592c06e71c4c528911f2095888a885 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:53:20.204795 23716 raft_consensus.cc:2454] T bd023ad546114d818753b5509d03bdcf P 6d09b4c117584c7b8ac25d3615f00d4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37a490e2b2c44e5d8d7b60a4af15312b in term 1.
I20260928 16:53:20.204798 23850 raft_consensus.cc:2454] T bd023ad546114d818753b5509d03bdcf P d0592c06e71c4c528911f2095888a885 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37a490e2b2c44e5d8d7b60a4af15312b in term 1.
I20260928 16:53:20.205324 23918 leader_election.cc:302] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37a490e2b2c44e5d8d7b60a4af15312b, 6d09b4c117584c7b8ac25d3615f00d4e; no voters:
I20260928 16:53:20.205664 24056 raft_consensus.cc:2781] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:53:20.206156 24056 raft_consensus.cc:686] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b [term 1 LEADER]: Becoming Leader. State: Replica: 37a490e2b2c44e5d8d7b60a4af15312b, State: Running, Role: LEADER
I20260928 16:53:20.206751 24056 consensus_queue.cc:237] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34841 } } peers { permanent_uuid: "6d09b4c117584c7b8ac25d3615f00d4e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38771 } } peers { permanent_uuid: "d0592c06e71c4c528911f2095888a885" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41983 } }
I20260928 16:53:20.212045 23571 catalog_manager.cc:5495] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b reported cstate change: term changed from 0 to 1, leader changed from <none> to 37a490e2b2c44e5d8d7b60a4af15312b (127.9.93.3). New cstate: current_term: 1 leader_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d09b4c117584c7b8ac25d3615f00d4e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0592c06e71c4c528911f2095888a885" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41983 } health_report { overall_health: UNKNOWN } } }
W20260928 16:53:20.223675 23896 tablet.cc:2369] T bd023ad546114d818753b5509d03bdcf P d0592c06e71c4c528911f2095888a885: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:53:20.230938 24044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:20.224019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47718 (local address 127.9.93.3:34841)
0928 16:53:20.224821 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:20.224826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:20.224843 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:53:20.224936 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:20.224940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:20.225038 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:20.225161 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:53:20.225649 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.226388 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.228618 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.229040 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.229139 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:20.229782 (+ 643us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:53:20.230381 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:53:20.230496 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:20.230621 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 16:53:20.230701 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665}
I20260928 16:53:20.241333 23716 raft_consensus.cc:1261] T bd023ad546114d818753b5509d03bdcf P 6d09b4c117584c7b8ac25d3615f00d4e [term 1 FOLLOWER]: Refusing update from remote peer 37a490e2b2c44e5d8d7b60a4af15312b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:53:20.241334 23850 raft_consensus.cc:1261] T bd023ad546114d818753b5509d03bdcf P d0592c06e71c4c528911f2095888a885 [term 1 FOLLOWER]: Refusing update from remote peer 37a490e2b2c44e5d8d7b60a4af15312b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:53:20.242319 24058 consensus_queue.cc:1035] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d09b4c117584c7b8ac25d3615f00d4e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:53:20.242687 24056 consensus_queue.cc:1035] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0592c06e71c4c528911f2095888a885" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:53:20.255666 24061 mvcc.cc:205] Tried to move back new op lower bound from 7334356583377289216 to 7334356583251558400. Current Snapshot: MvccSnapshot[applied={T|T < 7334356583377289216}]
I20260928 16:53:20.270931 23571 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:53:20.271389 23571 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:53:20.275564 23571 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25: Sending DeleteTablet for 3 replicas of tablet bd023ad546114d818753b5509d03bdcf
I20260928 16:53:20.276909 23964 tablet_service.cc:1514] Processing DeleteTablet for tablet bd023ad546114d818753b5509d03bdcf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:53:20 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:47716
I20260928 16:53:20.277467 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 23633
I20260928 16:53:20.277446 23696 tablet_service.cc:1514] Processing DeleteTablet for tablet bd023ad546114d818753b5509d03bdcf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:53:20 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:39120
I20260928 16:53:20.278244 23830 tablet_service.cc:1514] Processing DeleteTablet for tablet bd023ad546114d818753b5509d03bdcf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:53:20 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:41012
I20260928 16:53:20.300405 24074 tablet_replica.cc:330] T bd023ad546114d818753b5509d03bdcf P d0592c06e71c4c528911f2095888a885: stopping tablet replica
I20260928 16:53:20.301218 24072 tablet_replica.cc:330] T bd023ad546114d818753b5509d03bdcf P 6d09b4c117584c7b8ac25d3615f00d4e: stopping tablet replica
I20260928 16:53:20.301913 24074 raft_consensus.cc:2229] T bd023ad546114d818753b5509d03bdcf P d0592c06e71c4c528911f2095888a885 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:53:20.302639 24072 raft_consensus.cc:2229] T bd023ad546114d818753b5509d03bdcf P 6d09b4c117584c7b8ac25d3615f00d4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:53:20.303412 24074 raft_consensus.cc:2258] T bd023ad546114d818753b5509d03bdcf P d0592c06e71c4c528911f2095888a885 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:53:20.304024 24072 raft_consensus.cc:2258] T bd023ad546114d818753b5509d03bdcf P 6d09b4c117584c7b8ac25d3615f00d4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:53:20.307073 24074 ts_tablet_manager.cc:1905] T bd023ad546114d818753b5509d03bdcf P d0592c06e71c4c528911f2095888a885: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:53:20.317155 24071 tablet_replica.cc:330] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b: stopping tablet replica
I20260928 16:53:20.317332 24072 ts_tablet_manager.cc:1905] T bd023ad546114d818753b5509d03bdcf P 6d09b4c117584c7b8ac25d3615f00d4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:53:20.317785 24071 raft_consensus.cc:2229] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:53:20.318583 24071 raft_consensus.cc:2258] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:53:20.318869 24074 ts_tablet_manager.cc:1918] T bd023ad546114d818753b5509d03bdcf P d0592c06e71c4c528911f2095888a885: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 28 16:53:20 dist-test-slave-wzcb krb5kdc[23527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.1@KRBTEST.COM
I20260928 16:53:20.320767 24071 ts_tablet_manager.cc:1905] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:53:20.323206 24074 log.cc:1196] T bd023ad546114d818753b5509d03bdcf P d0592c06e71c4c528911f2095888a885: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/bd023ad546114d818753b5509d03bdcf
I20260928 16:53:20.323935 24074 ts_tablet_manager.cc:1939] T bd023ad546114d818753b5509d03bdcf P d0592c06e71c4c528911f2095888a885: Deleting consensus metadata
I20260928 16:53:20.325137 24072 ts_tablet_manager.cc:1918] T bd023ad546114d818753b5509d03bdcf P 6d09b4c117584c7b8ac25d3615f00d4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:53:20.325348 24072 log.cc:1196] T bd023ad546114d818753b5509d03bdcf P 6d09b4c117584c7b8ac25d3615f00d4e: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/wal/wals/bd023ad546114d818753b5509d03bdcf
I20260928 16:53:20.325954 24072 ts_tablet_manager.cc:1939] T bd023ad546114d818753b5509d03bdcf P 6d09b4c117584c7b8ac25d3615f00d4e: Deleting consensus metadata
I20260928 16:53:20.326297 23557 catalog_manager.cc:4841] TS d0592c06e71c4c528911f2095888a885 (127.9.93.2:41983): tablet bd023ad546114d818753b5509d03bdcf (table test-table [id=443c226734224cdd86a0974f82dbe77e]) successfully deleted
I20260928 16:53:20.328114 23557 catalog_manager.cc:4841] TS 6d09b4c117584c7b8ac25d3615f00d4e (127.9.93.1:38771): tablet bd023ad546114d818753b5509d03bdcf (table test-table [id=443c226734224cdd86a0974f82dbe77e]) successfully deleted
I20260928 16:53:20.330809 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:20.278825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39122 (local address 127.9.93.1:38771)
0928 16:53:20.279051 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:20.279059 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:20.308085 (+ 29026us) server_negotiation.cc:407] Connection header received
0928 16:53:20.308297 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:20.308303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:20.308417 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:20.308551 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:20.310930 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.312217 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.314550 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.315283 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.315427 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:20.321862 (+ 6435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:20.321892 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:20.321914 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:20.321970 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:20.326493 (+ 4523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:20.327698 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:20.327710 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:20.327714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:20.327921 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:20.328535 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:20.328541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:20.328545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:20.329091 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:20.329408 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:20.329920 (+ 512us) server_negotiation.cc:299] Negotiation successful
0928 16:53:20.330193 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 16:53:20.331597 23756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:20.332649 24071 ts_tablet_manager.cc:1918] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:53:20.332890 24071 log.cc:1196] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/bd023ad546114d818753b5509d03bdcf
I20260928 16:53:20.333575 24071 ts_tablet_manager.cc:1939] T bd023ad546114d818753b5509d03bdcf P 37a490e2b2c44e5d8d7b60a4af15312b: Deleting consensus metadata
I20260928 16:53:20.336134 23556 catalog_manager.cc:4841] TS 37a490e2b2c44e5d8d7b60a4af15312b (127.9.93.3:34841): tablet bd023ad546114d818753b5509d03bdcf (table test-table [id=443c226734224cdd86a0974f82dbe77e]) successfully deleted
I20260928 16:53:20.430766 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 23633
I20260928 16:53:20.453958 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 23767
Sep 28 16:53:20 dist-test-slave-wzcb krb5kdc[23527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.2@KRBTEST.COM
I20260928 16:53:20.480949 24047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:20.454954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41014 (local address 127.9.93.2:41983)
0928 16:53:20.455141 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:20.455146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:20.455211 (+ 65us) server_negotiation.cc:407] Connection header received
0928 16:53:20.455451 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:20.455455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:20.455540 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:20.455626 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:20.457413 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.458184 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.460279 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.460777 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.460888 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:20.472662 (+ 11774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:20.472680 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:20.472699 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:20.472764 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:20.477208 (+ 4444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:20.478292 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:20.478305 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:20.478309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:20.478492 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:20.478933 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:20.478938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:20.478941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:20.479631 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:20.479924 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:20.480332 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 16:53:20.480580 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 16:53:20.481725 23890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:20.572122 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 23767
I20260928 16:53:20.594789 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 23901
Sep 28 16:53:20 dist-test-slave-wzcb krb5kdc[23527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.3@KRBTEST.COM
I20260928 16:53:20.613582 24044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:20.595787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47720 (local address 127.9.93.3:34841)
0928 16:53:20.596006 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:20.596012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:20.596038 (+ 26us) server_negotiation.cc:407] Connection header received
0928 16:53:20.596108 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:20.596113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:20.596200 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:20.596287 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:20.597616 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.598508 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.600033 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:20.600532 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:20.600662 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:20.606008 (+ 5346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:20.606028 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:20.606051 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:20.606098 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:20.609727 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:20.610893 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:20.610902 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:20.610904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:20.611148 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:20.611771 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:20.611776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:20.611779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:20.612246 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:20.612541 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:20.612971 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 16:53:20.613178 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 16:53:20.614434 24024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:20.744796 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 23901
I20260928 16:53:20.762322 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 23540
I20260928 16:53:20.763764 23602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:20.881497 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 23540
I20260928 16:53:20.903849 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:42273
--webserver_interface=127.9.93.62
--webserver_port=43217
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:42273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:53:21.278612 24079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:21.278961 24079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:21.279083 24079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:21.286755 24079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:53:21.286900 24079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:21.286952 24079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:21.287055 24079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:53:21.287123 24079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:53:21.297868 24079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:42273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:42273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=43217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:21.299453 24079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:21.301000 24079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:21.328333 24086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:21.328456 24085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:21.328431 24088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:21.329448 24079 server_base.cc:1034] running on GCE node
I20260928 16:53:21.330451 24079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:21.332427 24079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:21.333678 24079 hybrid_clock.cc:648] HybridClock initialized: now 1790614401333613 us; error 50 us; skew 500 ppm
I20260928 16:53:21.334126 24079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:21 dist-test-slave-wzcb krb5kdc[23527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614401, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:21.338275 24079 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:53:21.340390 24079 webserver.cc:460] Webserver started at http://127.9.93.62:43217/ using document root <none> and password file <none>
I20260928 16:53:21.341266 24079 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:21.341403 24079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:53:21.348551 24079 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:53:21.352420 24095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:21.354314 24079 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:53:21.354543 24079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "68bb897b18f246eea07a972893292f25"
format_stamp: "Formatted at 2026-09-28 16:53:18 on dist-test-slave-wzcb"
server_key: "695157bd7fa2b586e02aa30321ed74c2"
server_key_iv: "502338b6206297f99a75949202791092"
server_key_version: "encryptionkey@0"
I20260928 16:53:21.355878 24079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:21.382155 24079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:21.387212 24079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:21.387566 24079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:21.407677 24079 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:42273
I20260928 16:53:21.407701 24146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:42273 every 8 connection(s)
I20260928 16:53:21.409454 24079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:53:21.412182 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 24079
I20260928 16:53:21.413252 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:38771
--local_ip_for_outbound_sockets=127.9.93.1
--tserver_master_addrs=127.9.93.62:42273
--webserver_port=33071
--webserver_interface=127.9.93.1
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:53:21.419097 24147 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:53:21.423055 24147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25: Bootstrap starting.
I20260928 16:53:21.445452 24147 log.cc:824] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25: Log is configured to *not* fsync() on all Append() calls
I20260928 16:53:21.459244 24147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:53:21.459918 24147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25: Bootstrap complete.
I20260928 16:53:21.470364 24147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bb897b18f246eea07a972893292f25" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42273 } }
I20260928 16:53:21.471423 24147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68bb897b18f246eea07a972893292f25, State: Initialized, Role: FOLLOWER
I20260928 16:53:21.472223 24147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bb897b18f246eea07a972893292f25" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42273 } }
I20260928 16:53:21.472498 24147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:53:21.472631 24147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:53:21.472816 24147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:53:21.477325 24147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bb897b18f246eea07a972893292f25" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42273 } }
I20260928 16:53:21.478026 24147 leader_election.cc:302] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68bb897b18f246eea07a972893292f25; no voters:
I20260928 16:53:21.478704 24147 leader_election.cc:288] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 16:53:21.478929 24151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 16:53:21.480157 24151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [term 2 LEADER]: Becoming Leader. State: Replica: 68bb897b18f246eea07a972893292f25, State: Running, Role: LEADER
I20260928 16:53:21.480846 24151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bb897b18f246eea07a972893292f25" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42273 } }
I20260928 16:53:21.481222 24147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:53:21.483536 24153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68bb897b18f246eea07a972893292f25. Latest consensus state: current_term: 2 leader_uuid: "68bb897b18f246eea07a972893292f25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bb897b18f246eea07a972893292f25" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42273 } } }
I20260928 16:53:21.483954 24153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [sys.catalog]: This master's current role is: LEADER
I20260928 16:53:21.484696 24152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "68bb897b18f246eea07a972893292f25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bb897b18f246eea07a972893292f25" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42273 } } }
I20260928 16:53:21.485008 24152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [sys.catalog]: This master's current role is: LEADER
I20260928 16:53:21.500108 24157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:53:21.509449 24157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:53:21.511196 24157 catalog_manager.cc:1259] Loaded cluster ID: f3a1f2102e814db08c4f8360a8290d0b
I20260928 16:53:21.511379 24157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:53:21.515769 24157 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 16:53:21.531879 24157 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 16:53:21.539956 24157 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:53:21.541510 24157 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25: Loaded TSK: 0
I20260928 16:53:21.542857 24157 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:53:21.831545 24149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:21.831888 24149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:21.832021 24149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:21.840039 24149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:21.840176 24149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:21.840329 24149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:53:21.851161 24149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:38771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=33071
--tserver_master_addrs=127.9.93.62:42273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:21.852821 24149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:21.854403 24149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:21.870018 24175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:21.869939 24176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:21.870617 24178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:21.871352 24149 server_base.cc:1034] running on GCE node
I20260928 16:53:21.872143 24149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:21.873353 24149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:21.874670 24149 hybrid_clock.cc:648] HybridClock initialized: now 1790614401874587 us; error 67 us; skew 500 ppm
I20260928 16:53:21.875108 24149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:21 dist-test-slave-wzcb krb5kdc[23527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614401, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:21.879050 24149 init.cc:377] Logged in from keytab as kudu/127.9.93.1@KRBTEST.COM (short username kudu)
I20260928 16:53:21.880946 24149 webserver.cc:460] Webserver started at http://127.9.93.1:33071/ using document root <none> and password file <none>
I20260928 16:53:21.881848 24149 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:21.882015 24149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:53:21.888778 24149 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 16:53:21.892323 24185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:21.894150 24149 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:53:21.894347 24149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "6d09b4c117584c7b8ac25d3615f00d4e"
format_stamp: "Formatted at 2026-09-28 16:53:18 on dist-test-slave-wzcb"
server_key: "f5281552e90275b15050134742f985bd"
server_key_iv: "a0f9355b7f632842857dd5a02bb1b869"
server_key_version: "encryptionkey@0"
I20260928 16:53:21.895794 24149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:21.929507 24149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:21.934262 24149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:21.934569 24149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:21.935768 24149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:53:21.937420 24149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:53:21.937518 24149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:21.937625 24149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:53:21.937696 24149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:21.971139 24149 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:38771
I20260928 16:53:21.971195 24297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:38771 every 8 connection(s)
I20260928 16:53:21.972821 24149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:53:21.983201 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 24149
I20260928 16:53:21.984078 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:41983
--local_ip_for_outbound_sockets=127.9.93.2
--tserver_master_addrs=127.9.93.62:42273
--webserver_port=36589
--webserver_interface=127.9.93.2
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:53:21 dist-test-slave-wzcb krb5kdc[23527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614401, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:53:22.002660 24301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:21.976269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.1:52525)
0928 16:53:21.977124 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:21.977171 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:21.978963 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:21.980749 (+ 1786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:21.980758 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:21.981470 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:53:21.982167 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:21.982193 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:21.985011 (+ 2818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:21.985018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:21.986528 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:21.986549 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:21.988215 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:21.988222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:21.988557 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:21.989396 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:53:21.989426 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:53:21.992398 (+ 2972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:53:21.997206 (+ 4808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:21.997215 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:21.997238 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:21.997798 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:21.998478 (+ 680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:21.998484 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:21.998487 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:21.998729 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:21.999922 (+ 1193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:53:21.999929 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:53:22.000486 (+ 557us) client_negotiation.cc:769] Sending connection context
0928 16:53:22.000835 (+ 349us) client_negotiation.cc:240] Negotiation successful
0928 16:53:22.001418 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":320,"threads_started":1}
I20260928 16:53:22.002787 24300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:21.976000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:52525 (local address 127.9.93.62:42273)
0928 16:53:21.976661 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:21.976681 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:21.977318 (+ 637us) server_negotiation.cc:407] Connection header received
0928 16:53:21.979667 (+ 2349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:21.979685 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:21.980119 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:21.980538 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:21.982465 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:21.984762 (+ 2297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:21.986739 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:21.987889 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:21.988110 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:21.992785 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:21.992795 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:21.992816 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:21.992887 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:21.996966 (+ 4079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:21.997975 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:21.997986 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:21.997995 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:21.998262 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:21.998869 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:21.998874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:21.998877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:21.999745 (+ 868us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:21.999940 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:22.001050 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0928 16:53:22.001629 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":238,"threads_started":1}
I20260928 16:53:22.006392 24298 heartbeater.cc:347] Connected to a master server at 127.9.93.62:42273
I20260928 16:53:22.006820 24298 heartbeater.cc:464] Registering TS with master...
I20260928 16:53:22.008023 24298 heartbeater.cc:511] Master 127.9.93.62:42273 requested a full tablet report, sending...
I20260928 16:53:22.011070 24112 ts_manager.cc:194] Registered new tserver with Master: 6d09b4c117584c7b8ac25d3615f00d4e (127.9.93.1:38771)
I20260928 16:53:22.012727 24112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.1@KRBTEST.COM'} at 127.9.93.1:52525
W20260928 16:53:22.349512 24302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:22.349848 24302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:22.349978 24302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:22.357599 24302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:22.357733 24302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:22.357897 24302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:53:22.368674 24302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:41983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=36589
--tserver_master_addrs=127.9.93.62:42273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:22.370188 24302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:22.371764 24302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:22.386281 24310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:22.386329 24307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:22.386358 24308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:22.387385 24302 server_base.cc:1034] running on GCE node
I20260928 16:53:22.388113 24302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:22.389187 24302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:22.390405 24302 hybrid_clock.cc:648] HybridClock initialized: now 1790614402390345 us; error 59 us; skew 500 ppm
I20260928 16:53:22.390786 24302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:22 dist-test-slave-wzcb krb5kdc[23527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614402, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:22.394477 24302 init.cc:377] Logged in from keytab as kudu/127.9.93.2@KRBTEST.COM (short username kudu)
I20260928 16:53:22.396217 24302 webserver.cc:460] Webserver started at http://127.9.93.2:36589/ using document root <none> and password file <none>
I20260928 16:53:22.397012 24302 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:22.397137 24302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:53:22.403610 24302 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 16:53:22.406880 24317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:22.408880 24302 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:53:22.409099 24302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "d0592c06e71c4c528911f2095888a885"
format_stamp: "Formatted at 2026-09-28 16:53:19 on dist-test-slave-wzcb"
server_key: "4aef89440d6683b3042ff3e0a2dbdc81"
server_key_iv: "eb327cd159e69a7d23bde1adc99c30ed"
server_key_version: "encryptionkey@0"
I20260928 16:53:22.410568 24302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:22.439687 24302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:22.444303 24302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:22.444594 24302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:22.445748 24302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:53:22.447432 24302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:53:22.447538 24302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:22.447655 24302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:53:22.447721 24302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:22.480960 24302 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:41983
I20260928 16:53:22.481041 24429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:41983 every 8 connection(s)
I20260928 16:53:22.482692 24302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:53:22.485962 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 24302
I20260928 16:53:22.486790 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:34841
--local_ip_for_outbound_sockets=127.9.93.3
--tserver_master_addrs=127.9.93.62:42273
--webserver_port=36985
--webserver_interface=127.9.93.3
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:53:22 dist-test-slave-wzcb krb5kdc[23527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614402, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:53:22.506899 24300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:22.485621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:57983 (local address 127.9.93.62:42273)
0928 16:53:22.485802 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:22.485806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:22.487474 (+ 1668us) server_negotiation.cc:407] Connection header received
0928 16:53:22.489567 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:22.489573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:22.489701 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:22.489793 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:22.491632 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:22.492386 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:22.494141 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:22.494612 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:22.494759 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:22.498819 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:22.498826 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:22.498832 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:22.498880 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:22.502549 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:22.503854 (+ 1305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:22.503860 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:22.503864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:22.504071 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:22.504685 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:22.504690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:22.504694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:22.505068 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:22.505225 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:22.506200 (+ 975us) server_negotiation.cc:299] Negotiation successful
0928 16:53:22.506508 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:53:22.507869 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:22.486304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.2:57983)
0928 16:53:22.487270 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:22.487341 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:22.489227 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:22.490115 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:22.490125 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:22.490731 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:53:22.491435 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:22.491459 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:22.492560 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:22.492564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:22.493957 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:22.493974 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:22.494890 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:22.494896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:22.495160 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:22.495998 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:53:22.496033 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:53:22.498522 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:53:22.502792 (+ 4270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:22.502805 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:22.502823 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:22.503658 (+ 835us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:22.504211 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:22.504216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:22.504218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:22.504536 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:22.505227 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:53:22.505235 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:53:22.505685 (+ 450us) client_negotiation.cc:769] Sending connection context
0928 16:53:22.506025 (+ 340us) client_negotiation.cc:240] Negotiation successful
0928 16:53:22.506506 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":347,"threads_started":1}
I20260928 16:53:22.509707 24430 heartbeater.cc:347] Connected to a master server at 127.9.93.62:42273
I20260928 16:53:22.510126 24430 heartbeater.cc:464] Registering TS with master...
I20260928 16:53:22.511196 24430 heartbeater.cc:511] Master 127.9.93.62:42273 requested a full tablet report, sending...
I20260928 16:53:22.513270 24112 ts_manager.cc:194] Registered new tserver with Master: d0592c06e71c4c528911f2095888a885 (127.9.93.2:41983)
I20260928 16:53:22.514218 24112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.2@KRBTEST.COM'} at 127.9.93.2:57983
W20260928 16:53:22.850168 24433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:22.850538 24433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:22.850674 24433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:22.858542 24433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:22.858707 24433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:22.858868 24433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:53:22.869875 24433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:34841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=36985
--tserver_master_addrs=127.9.93.62:42273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:22.871513 24433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:22.873149 24433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:22.887852 24438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:22.887768 24439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:22.887812 24441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:22.888731 24433 server_base.cc:1034] running on GCE node
I20260928 16:53:22.889469 24433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:22.890610 24433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:22.891841 24433 hybrid_clock.cc:648] HybridClock initialized: now 1790614402891782 us; error 52 us; skew 500 ppm
I20260928 16:53:22.892230 24433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:22 dist-test-slave-wzcb krb5kdc[23527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614402, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:22.895864 24433 init.cc:377] Logged in from keytab as kudu/127.9.93.3@KRBTEST.COM (short username kudu)
I20260928 16:53:22.897675 24433 webserver.cc:460] Webserver started at http://127.9.93.3:36985/ using document root <none> and password file <none>
I20260928 16:53:22.898514 24433 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:22.898666 24433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:53:22.905390 24433 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:53:22.908784 24448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:22.910609 24433 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:53:22.910812 24433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "37a490e2b2c44e5d8d7b60a4af15312b"
format_stamp: "Formatted at 2026-09-28 16:53:19 on dist-test-slave-wzcb"
server_key: "2c30f5ea8ad729eb78b56b18f154d733"
server_key_iv: "cfd49a00e49e987d9a23e9b4e97e37c2"
server_key_version: "encryptionkey@0"
I20260928 16:53:22.912153 24433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:22.932020 24433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:22.937012 24433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:22.937320 24433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:22.938458 24433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:53:22.939994 24433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:53:22.940114 24433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:22.940228 24433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:53:22.940299 24433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:22.974049 24433 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:34841
I20260928 16:53:22.974097 24560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:34841 every 8 connection(s)
I20260928 16:53:22.975904 24433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:53:22.986394 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 24433
Sep 28 16:53:22 dist-test-slave-wzcb krb5kdc[23527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614402, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:53:23.000025 24300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:22.978903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:33801 (local address 127.9.93.62:42273)
0928 16:53:22.979105 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:22.979109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:22.980761 (+ 1652us) server_negotiation.cc:407] Connection header received
0928 16:53:22.983017 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:22.983024 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:22.983129 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:22.983235 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:22.985238 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:22.986062 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:22.988330 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:22.988877 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:22.989029 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:22.993177 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:22.993183 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:22.993187 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:22.993232 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:22.996357 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:22.997274 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:22.997279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:22.997281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:22.997428 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:22.997927 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:22.997931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:22.997933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:22.998254 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:22.998418 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:22.999282 (+ 864us) server_negotiation.cc:299] Negotiation successful
0928 16:53:22.999656 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:53:23.001266 24563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:22.979539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.3:33801)
0928 16:53:22.980512 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:22.980580 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:22.982584 (+ 2004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:22.983609 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:22.983619 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:22.984279 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:53:22.984991 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:22.985014 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:22.986273 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:22.986277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:22.988094 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:22.988117 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:22.989175 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:22.989184 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:22.989469 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:22.990246 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:53:22.990273 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:53:22.992854 (+ 2581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:53:22.996549 (+ 3695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:22.996558 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:22.996576 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:22.997127 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:22.997550 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:22.997554 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:22.997556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:22.997786 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:22.998404 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:53:22.998413 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:53:22.998831 (+ 418us) client_negotiation.cc:769] Sending connection context
0928 16:53:22.999118 (+ 287us) client_negotiation.cc:240] Negotiation successful
0928 16:53:22.999803 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":376,"threads_started":1}
I20260928 16:53:23.003453 24561 heartbeater.cc:347] Connected to a master server at 127.9.93.62:42273
I20260928 16:53:23.003526 24565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:22.987468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53486 (local address 127.9.93.62:42273)
0928 16:53:22.988081 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:22.988088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:22.988677 (+ 589us) server_negotiation.cc:407] Connection header received
0928 16:53:22.988892 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:22.988897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:22.989005 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:22.989118 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:22.991051 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:22.991957 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:22.993500 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:22.994088 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:22.994236 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:22.997059 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:22.997064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:22.997068 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:22.997108 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:23.000161 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:23.001082 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:23.001088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:23.001091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:23.001260 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:23.001709 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:23.001713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:23.001716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:23.002088 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:23.002350 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:23.002749 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 16:53:23.003080 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":304,"threads_started":1}
I20260928 16:53:23.003808 24561 heartbeater.cc:464] Registering TS with master...
I20260928 16:53:23.004830 24561 heartbeater.cc:511] Master 127.9.93.62:42273 requested a full tablet report, sending...
I20260928 16:53:23.006654 24112 ts_manager.cc:194] Registered new tserver with Master: 37a490e2b2c44e5d8d7b60a4af15312b (127.9.93.3:34841)
I20260928 16:53:23.007597 24112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.3@KRBTEST.COM'} at 127.9.93.3:33801
I20260928 16:53:23.017123 24298 heartbeater.cc:503] Master 127.9.93.62:42273 was elected leader, sending a full tablet report...
I20260928 16:53:23.020210 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:53:23.028509 24565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:23.021299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53498 (local address 127.9.93.62:42273)
0928 16:53:23.021507 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:23.021512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:23.021949 (+ 437us) server_negotiation.cc:407] Connection header received
0928 16:53:23.022228 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:23.022233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:23.022459 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:23.022618 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:53:23.023092 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.023915 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.025706 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.026263 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.026393 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:23.027136 (+ 743us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:53:23.027886 (+ 750us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:53:23.028017 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:23.028178 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 16:53:23.028271 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:53:23.029973 24111 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:53498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:53:23.038553 24111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:53:23.078622 24574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:23.054816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54424 (local address 127.9.93.1:38771)
0928 16:53:23.065886 (+ 11070us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:23.065898 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:23.067903 (+ 2005us) server_negotiation.cc:407] Connection header received
0928 16:53:23.068327 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:23.068358 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:23.068695 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:23.068910 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:53:23.070169 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.072191 (+ 2022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.075862 (+ 3671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.077148 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.077274 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:23.077568 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:23.078068 (+ 500us) server_negotiation.cc:299] Negotiation successful
0928 16:53:23.078272 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10800,"thread_start_us":11118,"threads_started":1}
I20260928 16:53:23.078718 24571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:23.054383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:38771 (local address 127.0.0.1:54424)
0928 16:53:23.067693 (+ 13310us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:23.067751 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:23.067985 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:23.069337 (+ 1352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:23.069347 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:23.069385 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:53:23.069905 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:23.069920 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.072472 (+ 2552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.072477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:23.075595 (+ 3118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:23.075618 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.077501 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.077508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:23.077740 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:23.077800 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 16:53:23.077953 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 16:53:23.078205 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12986,"mutex_wait_us":1,"thread_start_us":12628,"threads_started":1}
I20260928 16:53:23.078881 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:23.054383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:34841 (local address 127.0.0.1:60908)
0928 16:53:23.068371 (+ 13988us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:23.068397 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:23.068682 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:23.069891 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:23.069897 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:23.069943 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:53:23.070392 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:23.070404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.072991 (+ 2587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.072997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:23.075775 (+ 2778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:23.075788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.078052 (+ 2264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.078059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:23.078220 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:23.078250 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 16:53:23.078337 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 16:53:23.078482 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13835,"mutex_wait_us":329,"thread_start_us":12839,"threads_started":1}
I20260928 16:53:23.079931 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:23.054605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60908 (local address 127.9.93.3:34841)
0928 16:53:23.066857 (+ 12252us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:23.066869 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:23.068525 (+ 1656us) server_negotiation.cc:407] Connection header received
0928 16:53:23.068932 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:23.068944 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:23.069255 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:23.069447 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:53:23.070649 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.072690 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.075980 (+ 3290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.077487 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.077632 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:23.077890 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:23.079355 (+ 1465us) server_negotiation.cc:299] Negotiation successful
0928 16:53:23.079568 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12022,"thread_start_us":11865,"threads_started":1}
I20260928 16:53:23.084358 24575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:23.068644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:41983 (local address 127.0.0.1:56066)
0928 16:53:23.075401 (+ 6757us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:23.075430 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:23.075588 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:23.076974 (+ 1386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:23.076979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:23.077016 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:53:23.077470 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:23.077484 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.079383 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.079387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:23.081281 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:23.081300 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.083463 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.083471 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:23.083686 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:23.083722 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 16:53:23.083835 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 16:53:23.083989 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6550,"thread_start_us":6716,"threads_started":1}
I20260928 16:53:23.084422 24576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:23.068639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56066 (local address 127.9.93.2:41983)
0928 16:53:23.076170 (+ 7531us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:23.076182 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:23.076214 (+ 32us) server_negotiation.cc:407] Connection header received
0928 16:53:23.076332 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:23.076342 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:23.076692 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:23.076951 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:53:23.077678 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.079172 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.081528 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.083149 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.083339 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:23.083655 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:23.083899 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 16:53:23.084086 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7328,"thread_start_us":7188,"threads_started":1}
I20260928 16:53:23.084308 24233 tablet_service.cc:1467] Processing CreateTablet for tablet df5ed96515574d5aacb89a8bf3445d40 (DEFAULT_TABLE table=test-table [id=b584b1c1963442628b022458e2cab95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:53:23.084308 24496 tablet_service.cc:1467] Processing CreateTablet for tablet df5ed96515574d5aacb89a8bf3445d40 (DEFAULT_TABLE table=test-table [id=b584b1c1963442628b022458e2cab95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:53:23.086207 24233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df5ed96515574d5aacb89a8bf3445d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:53:23.086261 24496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df5ed96515574d5aacb89a8bf3445d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:53:23.089074 24365 tablet_service.cc:1467] Processing CreateTablet for tablet df5ed96515574d5aacb89a8bf3445d40 (DEFAULT_TABLE table=test-table [id=b584b1c1963442628b022458e2cab95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:53:23.090826 24365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df5ed96515574d5aacb89a8bf3445d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:53:23.101361 24577 tablet_bootstrap.cc:492] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b: Bootstrap starting.
I20260928 16:53:23.105118 24578 tablet_bootstrap.cc:492] T df5ed96515574d5aacb89a8bf3445d40 P 6d09b4c117584c7b8ac25d3615f00d4e: Bootstrap starting.
I20260928 16:53:23.106098 24577 tablet_bootstrap.cc:654] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b: Neither blocks nor log segments found. Creating new log.
I20260928 16:53:23.107733 24577 log.cc:824] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b: Log is configured to *not* fsync() on all Append() calls
I20260928 16:53:23.109499 24578 tablet_bootstrap.cc:654] T df5ed96515574d5aacb89a8bf3445d40 P 6d09b4c117584c7b8ac25d3615f00d4e: Neither blocks nor log segments found. Creating new log.
I20260928 16:53:23.111505 24578 log.cc:824] T df5ed96515574d5aacb89a8bf3445d40 P 6d09b4c117584c7b8ac25d3615f00d4e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:53:23.114073 24580 tablet_bootstrap.cc:492] T df5ed96515574d5aacb89a8bf3445d40 P d0592c06e71c4c528911f2095888a885: Bootstrap starting.
I20260928 16:53:23.115622 24578 tablet_bootstrap.cc:492] T df5ed96515574d5aacb89a8bf3445d40 P 6d09b4c117584c7b8ac25d3615f00d4e: No bootstrap required, opened a new log
I20260928 16:53:23.115622 24577 tablet_bootstrap.cc:492] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b: No bootstrap required, opened a new log
I20260928 16:53:23.116010 24578 ts_tablet_manager.cc:1397] T df5ed96515574d5aacb89a8bf3445d40 P 6d09b4c117584c7b8ac25d3615f00d4e: Time spent bootstrapping tablet: real 0.011s user 0.011s sys 0.000s
I20260928 16:53:23.115998 24577 ts_tablet_manager.cc:1397] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20260928 16:53:23.118019 24580 tablet_bootstrap.cc:654] T df5ed96515574d5aacb89a8bf3445d40 P d0592c06e71c4c528911f2095888a885: Neither blocks nor log segments found. Creating new log.
I20260928 16:53:23.119246 24580 log.cc:824] T df5ed96515574d5aacb89a8bf3445d40 P d0592c06e71c4c528911f2095888a885: Log is configured to *not* fsync() on all Append() calls
I20260928 16:53:23.122394 24580 tablet_bootstrap.cc:492] T df5ed96515574d5aacb89a8bf3445d40 P d0592c06e71c4c528911f2095888a885: No bootstrap required, opened a new log
I20260928 16:53:23.122720 24580 ts_tablet_manager.cc:1397] T df5ed96515574d5aacb89a8bf3445d40 P d0592c06e71c4c528911f2095888a885: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 16:53:23.127848 24578 raft_consensus.cc:348] T df5ed96515574d5aacb89a8bf3445d40 P 6d09b4c117584c7b8ac25d3615f00d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34841 } } peers { permanent_uuid: "6d09b4c117584c7b8ac25d3615f00d4e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38771 } } peers { permanent_uuid: "d0592c06e71c4c528911f2095888a885" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41983 } }
I20260928 16:53:23.127720 24577 raft_consensus.cc:348] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34841 } } peers { permanent_uuid: "6d09b4c117584c7b8ac25d3615f00d4e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38771 } } peers { permanent_uuid: "d0592c06e71c4c528911f2095888a885" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41983 } }
I20260928 16:53:23.128324 24578 raft_consensus.cc:374] T df5ed96515574d5aacb89a8bf3445d40 P 6d09b4c117584c7b8ac25d3615f00d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:53:23.128316 24577 raft_consensus.cc:374] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:53:23.128433 24577 raft_consensus.cc:729] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a490e2b2c44e5d8d7b60a4af15312b, State: Initialized, Role: FOLLOWER
I20260928 16:53:23.128429 24578 raft_consensus.cc:729] T df5ed96515574d5aacb89a8bf3445d40 P 6d09b4c117584c7b8ac25d3615f00d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d09b4c117584c7b8ac25d3615f00d4e, State: Initialized, Role: FOLLOWER
I20260928 16:53:23.129199 24577 consensus_queue.cc:260] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34841 } } peers { permanent_uuid: "6d09b4c117584c7b8ac25d3615f00d4e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38771 } } peers { permanent_uuid: "d0592c06e71c4c528911f2095888a885" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41983 } }
I20260928 16:53:23.129200 24578 consensus_queue.cc:260] T df5ed96515574d5aacb89a8bf3445d40 P 6d09b4c117584c7b8ac25d3615f00d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34841 } } peers { permanent_uuid: "6d09b4c117584c7b8ac25d3615f00d4e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38771 } } peers { permanent_uuid: "d0592c06e71c4c528911f2095888a885" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41983 } }
I20260928 16:53:23.131148 24561 heartbeater.cc:503] Master 127.9.93.62:42273 was elected leader, sending a full tablet report...
I20260928 16:53:23.131855 24580 raft_consensus.cc:348] T df5ed96515574d5aacb89a8bf3445d40 P d0592c06e71c4c528911f2095888a885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34841 } } peers { permanent_uuid: "6d09b4c117584c7b8ac25d3615f00d4e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38771 } } peers { permanent_uuid: "d0592c06e71c4c528911f2095888a885" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41983 } }
I20260928 16:53:23.132416 24578 ts_tablet_manager.cc:1428] T df5ed96515574d5aacb89a8bf3445d40 P 6d09b4c117584c7b8ac25d3615f00d4e: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260928 16:53:23.132419 24580 raft_consensus.cc:374] T df5ed96515574d5aacb89a8bf3445d40 P d0592c06e71c4c528911f2095888a885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:53:23.132534 24580 raft_consensus.cc:729] T df5ed96515574d5aacb89a8bf3445d40 P d0592c06e71c4c528911f2095888a885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0592c06e71c4c528911f2095888a885, State: Initialized, Role: FOLLOWER
I20260928 16:53:23.133270 24580 consensus_queue.cc:260] T df5ed96515574d5aacb89a8bf3445d40 P d0592c06e71c4c528911f2095888a885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34841 } } peers { permanent_uuid: "6d09b4c117584c7b8ac25d3615f00d4e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38771 } } peers { permanent_uuid: "d0592c06e71c4c528911f2095888a885" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41983 } }
I20260928 16:53:23.134544 24577 ts_tablet_manager.cc:1428] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
I20260928 16:53:23.135428 24430 heartbeater.cc:503] Master 127.9.93.62:42273 was elected leader, sending a full tablet report...
I20260928 16:53:23.136165 24580 ts_tablet_manager.cc:1428] T df5ed96515574d5aacb89a8bf3445d40 P d0592c06e71c4c528911f2095888a885: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
W20260928 16:53:23.225481 24299 tablet.cc:2369] T df5ed96515574d5aacb89a8bf3445d40 P 6d09b4c117584c7b8ac25d3615f00d4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:53:23.227782 24562 tablet.cc:2369] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:53:23.234985 24431 tablet.cc:2369] T df5ed96515574d5aacb89a8bf3445d40 P d0592c06e71c4c528911f2095888a885: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:53:23.391121 24584 raft_consensus.cc:482] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:53:23.391578 24584 raft_consensus.cc:504] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34841 } } peers { permanent_uuid: "6d09b4c117584c7b8ac25d3615f00d4e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38771 } } peers { permanent_uuid: "d0592c06e71c4c528911f2095888a885" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41983 } }
I20260928 16:53:23.393559 24584 leader_election.cc:288] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d09b4c117584c7b8ac25d3615f00d4e (127.9.93.1:38771), d0592c06e71c4c528911f2095888a885 (127.9.93.2:41983)
I20260928 16:53:23.399763 24574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:23.394200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:36729 (local address 127.9.93.1:38771)
0928 16:53:23.394478 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:23.394485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:23.394510 (+ 25us) server_negotiation.cc:407] Connection header received
0928 16:53:23.394740 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:23.394744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:23.394811 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:23.394890 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:53:23.395349 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.396140 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.397904 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.398767 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.398903 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:23.399032 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:23.399294 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 16:53:23.399473 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 16:53:23.399756 24563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:23.394100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:38771 (local address 127.9.93.3:36729)
0928 16:53:23.394356 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:23.394379 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:23.394563 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:23.394928 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:23.394933 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:23.394960 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:53:23.395175 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:23.395185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.396308 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.396312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:23.397728 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:23.397745 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.398939 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.398945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:23.399133 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:23.399159 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:53:23.399246 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 16:53:23.399429 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 16:53:23.400950 24576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:23.394884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:32985 (local address 127.9.93.2:41983)
0928 16:53:23.395067 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:23.395074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:23.395560 (+ 486us) server_negotiation.cc:407] Connection header received
0928 16:53:23.395721 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:23.395725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:23.395790 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:23.395861 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:53:23.396404 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.397116 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.398885 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.399837 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.399990 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:23.400080 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:23.400549 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 16:53:23.400700 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 16:53:23.401038 24586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:23.394606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:41983 (local address 127.9.93.3:32985)
0928 16:53:23.395434 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:23.395452 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:23.395577 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:23.395893 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:23.395897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:23.395920 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:53:23.396232 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:23.396243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.397277 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.397281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:23.398710 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:23.398722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.400232 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.400236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:23.400378 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:23.400403 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 16:53:23.400510 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 16:53:23.400658 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":483,"threads_started":1}
I20260928 16:53:23.401067 24253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df5ed96515574d5aacb89a8bf3445d40" candidate_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d09b4c117584c7b8ac25d3615f00d4e" is_pre_election: true
I20260928 16:53:23.401732 24253 raft_consensus.cc:2454] T df5ed96515574d5aacb89a8bf3445d40 P 6d09b4c117584c7b8ac25d3615f00d4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37a490e2b2c44e5d8d7b60a4af15312b in term 0.
I20260928 16:53:23.402344 24385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df5ed96515574d5aacb89a8bf3445d40" candidate_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0592c06e71c4c528911f2095888a885" is_pre_election: true
I20260928 16:53:23.402541 24450 leader_election.cc:302] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37a490e2b2c44e5d8d7b60a4af15312b, 6d09b4c117584c7b8ac25d3615f00d4e; no voters:
I20260928 16:53:23.402840 24385 raft_consensus.cc:2454] T df5ed96515574d5aacb89a8bf3445d40 P d0592c06e71c4c528911f2095888a885 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37a490e2b2c44e5d8d7b60a4af15312b in term 0.
I20260928 16:53:23.402987 24584 raft_consensus.cc:2781] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:53:23.403139 24584 raft_consensus.cc:482] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:53:23.403254 24584 raft_consensus.cc:3037] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:53:23.405711 24584 raft_consensus.cc:504] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34841 } } peers { permanent_uuid: "6d09b4c117584c7b8ac25d3615f00d4e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38771 } } peers { permanent_uuid: "d0592c06e71c4c528911f2095888a885" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41983 } }
I20260928 16:53:23.406397 24584 leader_election.cc:288] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b [CANDIDATE]: Term 1 election: Requested vote from peers 6d09b4c117584c7b8ac25d3615f00d4e (127.9.93.1:38771), d0592c06e71c4c528911f2095888a885 (127.9.93.2:41983)
I20260928 16:53:23.407232 24385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df5ed96515574d5aacb89a8bf3445d40" candidate_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0592c06e71c4c528911f2095888a885"
I20260928 16:53:23.407251 24253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df5ed96515574d5aacb89a8bf3445d40" candidate_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d09b4c117584c7b8ac25d3615f00d4e"
I20260928 16:53:23.407552 24385 raft_consensus.cc:3037] T df5ed96515574d5aacb89a8bf3445d40 P d0592c06e71c4c528911f2095888a885 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:53:23.407555 24253 raft_consensus.cc:3037] T df5ed96515574d5aacb89a8bf3445d40 P 6d09b4c117584c7b8ac25d3615f00d4e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:53:23.411034 24385 raft_consensus.cc:2454] T df5ed96515574d5aacb89a8bf3445d40 P d0592c06e71c4c528911f2095888a885 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37a490e2b2c44e5d8d7b60a4af15312b in term 1.
I20260928 16:53:23.411029 24253 raft_consensus.cc:2454] T df5ed96515574d5aacb89a8bf3445d40 P 6d09b4c117584c7b8ac25d3615f00d4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37a490e2b2c44e5d8d7b60a4af15312b in term 1.
I20260928 16:53:23.411636 24450 leader_election.cc:302] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37a490e2b2c44e5d8d7b60a4af15312b, 6d09b4c117584c7b8ac25d3615f00d4e; no voters:
I20260928 16:53:23.412025 24584 raft_consensus.cc:2781] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:53:23.412499 24584 raft_consensus.cc:686] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b [term 1 LEADER]: Becoming Leader. State: Replica: 37a490e2b2c44e5d8d7b60a4af15312b, State: Running, Role: LEADER
I20260928 16:53:23.413103 24584 consensus_queue.cc:237] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34841 } } peers { permanent_uuid: "6d09b4c117584c7b8ac25d3615f00d4e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38771 } } peers { permanent_uuid: "d0592c06e71c4c528911f2095888a885" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41983 } }
I20260928 16:53:23.419087 24110 catalog_manager.cc:5495] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b reported cstate change: term changed from 0 to 1, leader changed from <none> to 37a490e2b2c44e5d8d7b60a4af15312b (127.9.93.3). New cstate: current_term: 1 leader_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a490e2b2c44e5d8d7b60a4af15312b" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d09b4c117584c7b8ac25d3615f00d4e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0592c06e71c4c528911f2095888a885" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41983 } health_report { overall_health: UNKNOWN } } }
I20260928 16:53:23.444069 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:23.438259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60922 (local address 127.9.93.3:34841)
0928 16:53:23.438566 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:23.438571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:23.438594 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:53:23.438762 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:23.438767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:23.438864 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:23.439018 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:53:23.439426 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.440160 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.441602 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.442051 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.442158 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:23.442759 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:53:23.443445 (+ 686us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:53:23.443580 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:23.443693 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 16:53:23.443778 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260928 16:53:23.454341 24385 raft_consensus.cc:1261] T df5ed96515574d5aacb89a8bf3445d40 P d0592c06e71c4c528911f2095888a885 [term 1 FOLLOWER]: Refusing update from remote peer 37a490e2b2c44e5d8d7b60a4af15312b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:53:23.454341 24253 raft_consensus.cc:1261] T df5ed96515574d5aacb89a8bf3445d40 P 6d09b4c117584c7b8ac25d3615f00d4e [term 1 FOLLOWER]: Refusing update from remote peer 37a490e2b2c44e5d8d7b60a4af15312b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:53:23.455577 24587 consensus_queue.cc:1035] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d09b4c117584c7b8ac25d3615f00d4e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:53:23.456068 24584 consensus_queue.cc:1035] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0592c06e71c4c528911f2095888a885" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 41983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:53:23.471524 24590 mvcc.cc:205] Tried to move back new op lower bound from 7334356596536487936 to 7334356596385153024. Current Snapshot: MvccSnapshot[applied={T|T < 7334356596536487936}]
I20260928 16:53:23.472374 24591 mvcc.cc:205] Tried to move back new op lower bound from 7334356596536487936 to 7334356596385153024. Current Snapshot: MvccSnapshot[applied={T|T < 7334356596536487936}]
I20260928 16:53:23.486445 24112 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:53498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:53:23.486783 24112 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:53498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:53:23.491379 24112 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25: Sending DeleteTablet for 3 replicas of tablet df5ed96515574d5aacb89a8bf3445d40
I20260928 16:53:23.492871 24233 tablet_service.cc:1514] Processing DeleteTablet for tablet df5ed96515574d5aacb89a8bf3445d40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:53:23 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:54424
I20260928 16:53:23.492877 24496 tablet_service.cc:1514] Processing DeleteTablet for tablet df5ed96515574d5aacb89a8bf3445d40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:53:23 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:60908
I20260928 16:53:23.495977 24600 tablet_replica.cc:330] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b: stopping tablet replica
I20260928 16:53:23.496392 24365 tablet_service.cc:1514] Processing DeleteTablet for tablet df5ed96515574d5aacb89a8bf3445d40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:53:23 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:56066
I20260928 16:53:23.496731 24599 tablet_replica.cc:330] T df5ed96515574d5aacb89a8bf3445d40 P 6d09b4c117584c7b8ac25d3615f00d4e: stopping tablet replica
I20260928 16:53:23.497355 24599 raft_consensus.cc:2229] T df5ed96515574d5aacb89a8bf3445d40 P 6d09b4c117584c7b8ac25d3615f00d4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:53:23.497941 24600 raft_consensus.cc:2229] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:53:23.498684 24599 raft_consensus.cc:2258] T df5ed96515574d5aacb89a8bf3445d40 P 6d09b4c117584c7b8ac25d3615f00d4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:53:23.499579 24600 raft_consensus.cc:2258] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:53:23.500451 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 24149
I20260928 16:53:23.502897 24599 ts_tablet_manager.cc:1905] T df5ed96515574d5aacb89a8bf3445d40 P 6d09b4c117584c7b8ac25d3615f00d4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:53:23.502903 24600 ts_tablet_manager.cc:1905] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:53:23.511019 24599 ts_tablet_manager.cc:1918] T df5ed96515574d5aacb89a8bf3445d40 P 6d09b4c117584c7b8ac25d3615f00d4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:53:23.511430 24600 ts_tablet_manager.cc:1918] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:53:23.511721 24600 log.cc:1196] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/df5ed96515574d5aacb89a8bf3445d40
I20260928 16:53:23.512486 24600 ts_tablet_manager.cc:1939] T df5ed96515574d5aacb89a8bf3445d40 P 37a490e2b2c44e5d8d7b60a4af15312b: Deleting consensus metadata
I20260928 16:53:23.514832 24096 catalog_manager.cc:4841] TS 37a490e2b2c44e5d8d7b60a4af15312b (127.9.93.3:34841): tablet df5ed96515574d5aacb89a8bf3445d40 (table test-table [id=b584b1c1963442628b022458e2cab95d]) successfully deleted
I20260928 16:53:23.511271 24599 log.cc:1196] T df5ed96515574d5aacb89a8bf3445d40 P 6d09b4c117584c7b8ac25d3615f00d4e: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/wal/wals/df5ed96515574d5aacb89a8bf3445d40
I20260928 16:53:23.516484 24599 ts_tablet_manager.cc:1939] T df5ed96515574d5aacb89a8bf3445d40 P 6d09b4c117584c7b8ac25d3615f00d4e: Deleting consensus metadata
I20260928 16:53:23.519228 24097 catalog_manager.cc:4841] TS 6d09b4c117584c7b8ac25d3615f00d4e (127.9.93.1:38771): tablet df5ed96515574d5aacb89a8bf3445d40 (table test-table [id=b584b1c1963442628b022458e2cab95d]) successfully deleted
I20260928 16:53:23.521649 24601 tablet_replica.cc:330] T df5ed96515574d5aacb89a8bf3445d40 P d0592c06e71c4c528911f2095888a885: stopping tablet replica
I20260928 16:53:23.522096 24601 raft_consensus.cc:2229] T df5ed96515574d5aacb89a8bf3445d40 P d0592c06e71c4c528911f2095888a885 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:53:23.522733 24601 raft_consensus.cc:2258] T df5ed96515574d5aacb89a8bf3445d40 P d0592c06e71c4c528911f2095888a885 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:53:23.523651 24574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:23.502077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54440 (local address 127.9.93.1:38771)
0928 16:53:23.502260 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:23.502265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:23.502285 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:53:23.502352 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:23.502357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:23.502452 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:23.502563 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:23.504639 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.505801 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.507698 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.508412 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.508559 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:23.514381 (+ 5822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:23.514406 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:23.514430 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:23.514496 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:23.519386 (+ 4890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:23.520811 (+ 1425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:23.520820 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:23.520822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:23.520989 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:23.521548 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:23.521555 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:23.521559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:23.522133 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:23.522484 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:23.522909 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 16:53:23.523202 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 16:53:23.524366 24293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:23.525804 24601 ts_tablet_manager.cc:1905] T df5ed96515574d5aacb89a8bf3445d40 P d0592c06e71c4c528911f2095888a885: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:53:23.533908 24601 ts_tablet_manager.cc:1918] T df5ed96515574d5aacb89a8bf3445d40 P d0592c06e71c4c528911f2095888a885: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:53:23.534161 24601 log.cc:1196] T df5ed96515574d5aacb89a8bf3445d40 P d0592c06e71c4c528911f2095888a885: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/df5ed96515574d5aacb89a8bf3445d40
I20260928 16:53:23.534788 24601 ts_tablet_manager.cc:1939] T df5ed96515574d5aacb89a8bf3445d40 P d0592c06e71c4c528911f2095888a885: Deleting consensus metadata
I20260928 16:53:23.537231 24097 catalog_manager.cc:4841] TS d0592c06e71c4c528911f2095888a885 (127.9.93.2:41983): tablet df5ed96515574d5aacb89a8bf3445d40 (table test-table [id=b584b1c1963442628b022458e2cab95d]) successfully deleted
I20260928 16:53:23.616786 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 24149
I20260928 16:53:23.634644 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 24302
I20260928 16:53:23.652971 24576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:23.635569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56082 (local address 127.9.93.2:41983)
0928 16:53:23.635778 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:23.635786 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:23.635821 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:53:23.636021 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:23.636027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:23.636142 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:23.636270 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:23.637839 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.638872 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.640362 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.640997 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.641139 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:23.644747 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:23.644774 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:23.644800 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:23.644872 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:23.649027 (+ 4155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:23.650127 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:23.650140 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:23.650144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:23.650368 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:23.650912 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:23.650918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:23.650921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:23.651528 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:23.651877 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:23.652259 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 16:53:23.652539 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 16:53:23.653891 24425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:23.747176 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 24302
I20260928 16:53:23.764032 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 24433
I20260928 16:53:23.780632 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:23.765023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60924 (local address 127.9.93.3:34841)
0928 16:53:23.765229 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:23.765234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:23.765284 (+ 50us) server_negotiation.cc:407] Connection header received
0928 16:53:23.765536 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:23.765541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:23.765621 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:23.765718 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:23.767253 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.768273 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.769816 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:23.770316 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:23.770439 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:23.773536 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:23.773557 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:23.773576 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:23.773637 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:23.777045 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:23.778020 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:23.778037 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:23.778039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:23.778271 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:23.778761 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:23.778765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:23.778767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:23.779196 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:23.779631 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:23.779940 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 16:53:23.780196 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 16:53:23.781533 24556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:23.909381 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 24433
I20260928 16:53:23.927021 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 24079
I20260928 16:53:23.928364 24142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:24.048843 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 24079
I20260928 16:53:24.070724 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:42273
--webserver_interface=127.9.93.62
--webserver_port=43217
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:42273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:53:24.437999 24606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:24.438388 24606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:24.438534 24606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:24.447093 24606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:53:24.447338 24606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:24.447455 24606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:24.447553 24606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:53:24.447626 24606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:53:24.458844 24606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:42273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:42273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=43217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:24.460575 24606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:24.462178 24606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:24.476327 24614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:24.476468 24612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:24.476436 24611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:24.477800 24606 server_base.cc:1034] running on GCE node
I20260928 16:53:24.478765 24606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:24.480631 24606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:24.481904 24606 hybrid_clock.cc:648] HybridClock initialized: now 1790614404481823 us; error 58 us; skew 500 ppm
I20260928 16:53:24.482314 24606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:24 dist-test-slave-wzcb krb5kdc[23527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614404, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:24.491098 24606 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:53:24.493052 24606 webserver.cc:460] Webserver started at http://127.9.93.62:43217/ using document root <none> and password file <none>
I20260928 16:53:24.493902 24606 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:24.494050 24606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:53:24.500746 24606 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:53:24.504191 24621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:24.505776 24606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:53:24.505990 24606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "68bb897b18f246eea07a972893292f25"
format_stamp: "Formatted at 2026-09-28 16:53:18 on dist-test-slave-wzcb"
server_key: "695157bd7fa2b586e02aa30321ed74c2"
server_key_iv: "502338b6206297f99a75949202791092"
server_key_version: "encryptionkey@0"
I20260928 16:53:24.507288 24606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:24.525187 24606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:24.530404 24606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:24.530722 24606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:24.551607 24606 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:42273
I20260928 16:53:24.551625 24672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:42273 every 8 connection(s)
I20260928 16:53:24.553639 24606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:53:24.558658 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 24606
I20260928 16:53:24.559743 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:38771
--local_ip_for_outbound_sockets=127.9.93.1
--tserver_master_addrs=127.9.93.62:42273
--webserver_port=33071
--webserver_interface=127.9.93.1
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:53:24.563007 24673 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:53:24.568440 24673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25: Bootstrap starting.
I20260928 16:53:24.591931 24673 log.cc:824] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25: Log is configured to *not* fsync() on all Append() calls
I20260928 16:53:24.610531 24673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:53:24.611232 24673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25: Bootstrap complete.
I20260928 16:53:24.621347 24673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bb897b18f246eea07a972893292f25" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42273 } }
I20260928 16:53:24.622421 24673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68bb897b18f246eea07a972893292f25, State: Initialized, Role: FOLLOWER
I20260928 16:53:24.623123 24673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bb897b18f246eea07a972893292f25" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42273 } }
I20260928 16:53:24.623399 24673 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:53:24.623529 24673 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:53:24.623708 24673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [term 2 FOLLOWER]: Advancing to term 3
I20260928 16:53:24.628196 24673 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bb897b18f246eea07a972893292f25" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42273 } }
I20260928 16:53:24.628786 24673 leader_election.cc:302] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68bb897b18f246eea07a972893292f25; no voters:
I20260928 16:53:24.629489 24673 leader_election.cc:288] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 16:53:24.629647 24677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 16:53:24.630761 24677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [term 3 LEADER]: Becoming Leader. State: Replica: 68bb897b18f246eea07a972893292f25, State: Running, Role: LEADER
I20260928 16:53:24.631453 24677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bb897b18f246eea07a972893292f25" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42273 } }
I20260928 16:53:24.632011 24673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:53:24.634481 24678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "68bb897b18f246eea07a972893292f25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bb897b18f246eea07a972893292f25" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42273 } } }
I20260928 16:53:24.634964 24678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [sys.catalog]: This master's current role is: LEADER
I20260928 16:53:24.634514 24679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68bb897b18f246eea07a972893292f25. Latest consensus state: current_term: 3 leader_uuid: "68bb897b18f246eea07a972893292f25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bb897b18f246eea07a972893292f25" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42273 } } }
I20260928 16:53:24.635934 24679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bb897b18f246eea07a972893292f25 [sys.catalog]: This master's current role is: LEADER
I20260928 16:53:24.647936 24683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:53:24.658568 24683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:53:24.660503 24683 catalog_manager.cc:1259] Loaded cluster ID: f3a1f2102e814db08c4f8360a8290d0b
I20260928 16:53:24.660650 24683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 16:53:24.663197 24683 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9ed4ada9ad google::LogMessage::Fail() at ??:0
@ 0x7f9ed4add88a google::LogMessage::SendToLog() at ??:0
@ 0x7f9ed4ada4de google::LogMessage::Flush() at ??:0
@ 0x7f9ed4adba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9eea50d512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f9eea50a976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9eea592610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f9eea592439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9ed7726345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9ed772991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f9ed7729799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9ed7704eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9eda9de6db start_thread at ??:0
@ 0x7f9ed397461f clone at ??:0
W20260928 16:53:25.006772 24675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:25.007184 24675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:25.007360 24675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:25.015821 24675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:25.015980 24675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:25.016161 24675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:53:25.027073 24675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:38771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=33071
--tserver_master_addrs=127.9.93.62:42273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:25.028741 24675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:25.030665 24675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:25.046676 24715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:25.046766 24717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:25.046684 24714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:25.047647 24675 server_base.cc:1034] running on GCE node
I20260928 16:53:25.048463 24675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:25.049686 24675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:25.050961 24675 hybrid_clock.cc:648] HybridClock initialized: now 1790614405050920 us; error 99 us; skew 500 ppm
I20260928 16:53:25.051481 24675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:25 dist-test-slave-wzcb krb5kdc[23527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614405, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:25.055469 24675 init.cc:377] Logged in from keytab as kudu/127.9.93.1@KRBTEST.COM (short username kudu)
I20260928 16:53:25.057322 24675 webserver.cc:460] Webserver started at http://127.9.93.1:33071/ using document root <none> and password file <none>
I20260928 16:53:25.058197 24675 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:25.058339 24675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:53:25.065632 24675 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:53:25.069425 24724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:25.071504 24675 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:53:25.071734 24675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "6d09b4c117584c7b8ac25d3615f00d4e"
format_stamp: "Formatted at 2026-09-28 16:53:18 on dist-test-slave-wzcb"
server_key: "f5281552e90275b15050134742f985bd"
server_key_iv: "a0f9355b7f632842857dd5a02bb1b869"
server_key_version: "encryptionkey@0"
I20260928 16:53:25.072989 24675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:25.093343 24675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:25.098390 24675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:25.098706 24675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:25.100011 24675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:53:25.101694 24675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:53:25.101830 24675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:25.101962 24675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:53:25.102051 24675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:25.138121 24675 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:38771
I20260928 16:53:25.138180 24836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:38771 every 8 connection(s)
I20260928 16:53:25.139925 24675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:53:25.143040 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 24675
I20260928 16:53:25.143751 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:41983
--local_ip_for_outbound_sockets=127.9.93.2
--tserver_master_addrs=127.9.93.62:42273
--webserver_port=36589
--webserver_interface=127.9.93.2
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:53:25.146174 24839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:25.143610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.1:38717)
0928 16:53:25.144637 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:25.144929 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":438,"threads_started":1}
W20260928 16:53:25.147125 24837 heartbeater.cc:644] Failed to heartbeat to 127.9.93.62:42273 (0 consecutive failures): Network error: Failed to ping master at 127.9.93.62:42273: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
I20260928 16:53:25.148313 24839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:25.147873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.1:52205)
0928 16:53:25.148073 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:25.148160 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 16:53:25.149326 24839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:25.148935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.1:42879)
0928 16:53:25.149110 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:25.149197 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 16:53:25.149586 24837 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:53:25.522379 24840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:25.522708 24840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:25.522838 24840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:25.530540 24840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:25.530675 24840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:25.530848 24840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:53:25.541338 24840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:41983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=36589
--tserver_master_addrs=127.9.93.62:42273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:25.542889 24840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:25.544555 24840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:25.559965 24845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:25.560011 24848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:25.560032 24846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:25.560739 24840 server_base.cc:1034] running on GCE node
I20260928 16:53:25.561487 24840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:25.562708 24840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:25.563937 24840 hybrid_clock.cc:648] HybridClock initialized: now 1790614405563864 us; error 61 us; skew 500 ppm
I20260928 16:53:25.564297 24840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:25 dist-test-slave-wzcb krb5kdc[23527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614405, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:25.567863 24840 init.cc:377] Logged in from keytab as kudu/127.9.93.2@KRBTEST.COM (short username kudu)
I20260928 16:53:25.569620 24840 webserver.cc:460] Webserver started at http://127.9.93.2:36589/ using document root <none> and password file <none>
I20260928 16:53:25.570477 24840 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:25.570600 24840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:53:25.576992 24840 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:53:25.580456 24855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:25.582269 24840 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:53:25.582468 24840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "d0592c06e71c4c528911f2095888a885"
format_stamp: "Formatted at 2026-09-28 16:53:19 on dist-test-slave-wzcb"
server_key: "4aef89440d6683b3042ff3e0a2dbdc81"
server_key_iv: "eb327cd159e69a7d23bde1adc99c30ed"
server_key_version: "encryptionkey@0"
I20260928 16:53:25.583664 24840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:25.602556 24840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:25.607506 24840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:25.607774 24840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:25.608902 24840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:53:25.610396 24840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:53:25.610502 24840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:25.610592 24840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:53:25.610647 24840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:25.644825 24840 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:41983
I20260928 16:53:25.644884 24967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:41983 every 8 connection(s)
I20260928 16:53:25.646483 24840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:53:25.652289 24970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:25.650075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.2:43415)
0928 16:53:25.650911 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:25.651157 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":317,"threads_started":1}
I20260928 16:53:25.653092 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 24840
W20260928 16:53:25.653194 24968 heartbeater.cc:644] Failed to heartbeat to 127.9.93.62:42273 (0 consecutive failures): Network error: Failed to ping master at 127.9.93.62:42273: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
I20260928 16:53:25.653852 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:34841
--local_ip_for_outbound_sockets=127.9.93.3
--tserver_master_addrs=127.9.93.62:42273
--webserver_port=36985
--webserver_interface=127.9.93.3
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:53:25.654304 24970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:25.653842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.2:46199)
0928 16:53:25.654059 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:25.654155 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 16:53:25.655404 24970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:25.654942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.2:38091)
0928 16:53:25.655112 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:25.655215 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 16:53:25.655711 24968 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:53:26.016144 24971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:26.016475 24971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:26.016640 24971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:26.024279 24971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:26.024425 24971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:26.024588 24971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:53:26.035049 24971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:34841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=36985
--tserver_master_addrs=127.9.93.62:42273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:26.036599 24971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:26.038102 24971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:26.053211 24976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:26.053210 24979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:26.053239 24977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:26.054239 24971 server_base.cc:1034] running on GCE node
I20260928 16:53:26.054977 24971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:26.056141 24971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:26.057371 24971 hybrid_clock.cc:648] HybridClock initialized: now 1790614406057325 us; error 73 us; skew 500 ppm
I20260928 16:53:26.057749 24971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:26 dist-test-slave-wzcb krb5kdc[23527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614406, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:26.061326 24971 init.cc:377] Logged in from keytab as kudu/127.9.93.3@KRBTEST.COM (short username kudu)
I20260928 16:53:26.063097 24971 webserver.cc:460] Webserver started at http://127.9.93.3:36985/ using document root <none> and password file <none>
I20260928 16:53:26.063979 24971 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:26.064116 24971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:53:26.070829 24971 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:53:26.078680 24986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:26.081148 24971 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.003s
I20260928 16:53:26.081403 24971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "37a490e2b2c44e5d8d7b60a4af15312b"
format_stamp: "Formatted at 2026-09-28 16:53:19 on dist-test-slave-wzcb"
server_key: "2c30f5ea8ad729eb78b56b18f154d733"
server_key_iv: "cfd49a00e49e987d9a23e9b4e97e37c2"
server_key_version: "encryptionkey@0"
I20260928 16:53:26.082618 24971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:26.106537 24971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:26.111783 24971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:26.112100 24971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:26.113318 24971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:53:26.115001 24971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:53:26.115118 24971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:26.115248 24971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:53:26.115350 24971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:26.149654 24971 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:34841
I20260928 16:53:26.149726 25098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:34841 every 8 connection(s)
I20260928 16:53:26.151288 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:26.150247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.1:48081)
0928 16:53:26.150818 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:26.150945 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":244,"threads_started":1}
I20260928 16:53:26.151795 24971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:53:26.152828 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 24971
I20260928 16:53:26.157825 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:26.155636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.3:55741)
0928 16:53:26.156429 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:26.156668 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":296,"threads_started":1}
W20260928 16:53:26.158618 25100 heartbeater.cc:644] Failed to heartbeat to 127.9.93.62:42273 (0 consecutive failures): Network error: Failed to ping master at 127.9.93.62:42273: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
I20260928 16:53:26.159693 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:26.159278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.3:38471)
0928 16:53:26.159482 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:26.159566 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 16:53:26.160632 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:26.160274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.3:35293)
0928 16:53:26.160412 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:26.160494 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 16:53:26.160890 25100 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 16:53:26.657526 25104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:26.656537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.2:58277)
0928 16:53:26.657107 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:26.657226 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":264,"threads_started":1}
I20260928 16:53:27.153375 25105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:27.152484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.1:38109)
0928 16:53:27.153038 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:27.153150 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":266,"threads_started":1}
I20260928 16:53:27.162489 25106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:27.161560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.3:48559)
0928 16:53:27.162118 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:27.162219 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":288,"threads_started":1}
I20260928 16:53:27.659626 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:27.658672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.2:38481)
0928 16:53:27.659221 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:27.659429 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":254,"threads_started":1}
I20260928 16:53:28.155448 25108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:28.154451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.1:39481)
0928 16:53:28.155071 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:28.155174 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":319,"threads_started":1}
I20260928 16:53:28.164350 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:28.163551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.3:43253)
0928 16:53:28.164078 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:28.164168 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":269,"threads_started":1}
I20260928 16:53:28.661872 25110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:28.660860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.2:45187)
0928 16:53:28.661475 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:28.661607 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":298,"threads_started":1}
I20260928 16:53:29.157472 25111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:29.156582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.1:35785)
0928 16:53:29.157174 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:29.157278 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":282,"threads_started":1}
I20260928 16:53:29.166119 25112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:29.165278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.3:44369)
0928 16:53:29.165805 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:29.165896 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":296,"threads_started":1}
I20260928 16:53:29.664041 25113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:29.662998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.2:56963)
0928 16:53:29.663705 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:29.663837 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":389,"threads_started":1}
I20260928 16:53:30.159704 25114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:30.158698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.1:42695)
0928 16:53:30.159374 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:30.159498 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":332,"threads_started":1}
I20260928 16:53:30.168071 25115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:30.167188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.3:40101)
0928 16:53:30.167764 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:30.167859 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":314,"threads_started":1}
I20260928 16:53:30.666056 25116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:30.665144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.2:52099)
0928 16:53:30.665746 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:30.665842 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":336,"threads_started":1}
I20260928 16:53:31.161670 25117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:31.160792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.1:41523)
0928 16:53:31.161374 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:31.161488 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":309,"threads_started":1}
I20260928 16:53:31.170318 25118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:31.169184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.3:34467)
0928 16:53:31.169991 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:31.170114 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":320,"threads_started":1}
I20260928 16:53:31.667972 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:31.667108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.2:37873)
0928 16:53:31.667670 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:31.667792 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":313,"threads_started":1}
I20260928 16:53:32.163861 25120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:32.162774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.1:57189)
0928 16:53:32.163478 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:32.163597 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":273,"threads_started":1}
I20260928 16:53:32.172501 25121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:32.171577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.3:57215)
0928 16:53:32.172171 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:32.172314 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":325,"threads_started":1}
I20260928 16:53:32.670149 25122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:32.669106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.2:39209)
0928 16:53:32.669794 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:32.669897 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":302,"threads_started":1}
I20260928 16:53:33.165941 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:33.165055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.1:60103)
0928 16:53:33.165655 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:33.165747 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":346,"threads_started":1}
I20260928 16:53:33.174568 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:33.173624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.3:42143)
0928 16:53:33.174238 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:33.174336 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":246,"threads_started":1}
I20260928 16:53:33.672035 25125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:33.671196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.2:59977)
0928 16:53:33.671733 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:33.671835 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":289,"threads_started":1}
I20260928 16:53:34.168000 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:34.167053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.1:49359)
0928 16:53:34.167690 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:34.167794 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":334,"threads_started":1}
I20260928 16:53:34.176311 25127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:34.175601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.3:34387)
0928 16:53:34.176047 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:34.176137 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":207,"threads_started":1}
I20260928 16:53:34.674036 25128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:34.673149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.2:58905)
0928 16:53:34.673728 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:34.673834 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":290,"threads_started":1}
I20260928 16:53:35.170137 25129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:35.169216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.1:40389)
0928 16:53:35.169834 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:35.169945 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":344,"threads_started":1}
I20260928 16:53:35.178226 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:35.177289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.3:53871)
0928 16:53:35.177919 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:35.178029 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":321,"threads_started":1}
I20260928 16:53:35.676044 25131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:35.675191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.2:43429)
0928 16:53:35.675752 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:35.675852 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":313,"threads_started":1}
I20260928 16:53:36.172363 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:36.171403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.1:52509)
0928 16:53:36.171990 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:36.172119 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":291,"threads_started":1}
I20260928 16:53:36.180284 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:36.179464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.3:43155)
0928 16:53:36.179980 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:36.180087 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":279,"threads_started":1}
I20260928 16:53:36.678009 25134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:36.677150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.2:45791)
0928 16:53:36.677660 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:36.677774 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":237,"threads_started":1}
I20260928 16:53:37.174356 25135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:37.173476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.1:58807)
0928 16:53:37.174052 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:37.174177 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":252,"threads_started":1}
I20260928 16:53:37.182215 25136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:37.181421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.3:39059)
0928 16:53:37.181915 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:37.182024 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":234,"threads_started":1}
I20260928 16:53:37.680032 25137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:37.679116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.2:55209)
0928 16:53:37.679713 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:37.679832 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":314,"threads_started":1}
I20260928 16:53:38.176710 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:38.175703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.1:57025)
0928 16:53:38.176271 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:38.176396 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":269,"threads_started":1}
I20260928 16:53:38.184069 25139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:38.183253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.3:40435)
0928 16:53:38.183786 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:38.183899 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":298,"threads_started":1}
I20260928 16:53:38.683248 25140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:38.681402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.2:51363)
0928 16:53:38.682638 (+ 1236us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:38.682962 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1074,"thread_start_us":297,"threads_started":1}
I20260928 16:53:39.178669 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:39.177729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.1:45383)
0928 16:53:39.178376 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:39.178468 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":312,"threads_started":1}
I20260928 16:53:39.186005 25142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:39.185128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.3:53845)
0928 16:53:39.185641 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:39.185772 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":220,"threads_started":1}
I20260928 16:53:39.685489 25143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:39.684464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.2:46185)
0928 16:53:39.685101 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:39.685228 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":305,"threads_started":1}
I20260928 16:53:40.180986 25144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:40.180116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.1:46915)
0928 16:53:40.180667 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:40.180769 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":290,"threads_started":1}
I20260928 16:53:40.188125 25145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:40.187261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.3:44697)
0928 16:53:40.187784 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:40.187881 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":247,"threads_started":1}
I20260928 16:53:40.687990 25146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:40.686941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.2:38819)
0928 16:53:40.687621 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:40.687771 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":284,"threads_started":1}
I20260928 16:53:41.160583 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 24675
I20260928 16:53:41.183288 25148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:41.182231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.1:53635)
0928 16:53:41.182828 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:41.182992 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":306,"threads_started":1}
I20260928 16:53:41.185819 25147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:41.161994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.9.93.1:38771)
0928 16:53:41.162806 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:41.162828 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:41.162887 (+ 59us) server_negotiation.cc:407] Connection header received
0928 16:53:41.165273 (+ 2386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:41.165288 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:41.165857 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:41.166240 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:41.168132 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:41.169679 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:41.171482 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:41.172467 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:41.172691 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:41.176244 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:41.176255 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:41.176279 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:41.176339 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:41.180856 (+ 4517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:41.182147 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:41.182157 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:41.182165 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:41.182449 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:41.183028 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:41.183033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:41.183035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:41.183950 (+ 915us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:41.184314 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:41.184910 (+ 596us) server_negotiation.cc:299] Negotiation successful
0928 16:53:41.185490 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":446,"threads_started":1}
I20260928 16:53:41.187619 24832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:41.190392 25150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:41.189422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:42273 (local address 127.9.93.3:60163)
0928 16:53:41.189981 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:41.190116 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":286,"threads_started":1}
I20260928 16:53:41.281502 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 24675
I20260928 16:53:41.300246 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 24840
I20260928 16:53:41.321519 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:41.301452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53150 (local address 127.9.93.2:41983)
0928 16:53:41.302064 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:41.302081 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:41.302125 (+ 44us) server_negotiation.cc:407] Connection header received
0928 16:53:41.303870 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:41.303889 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:41.304414 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:41.304826 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:41.306524 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:41.308102 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:41.309648 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:41.310486 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:41.310703 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:41.313860 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:41.313869 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:41.313891 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:41.313946 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:41.317368 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:41.318271 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:41.318280 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:41.318289 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:41.318529 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:41.318980 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:41.318984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:41.318986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:41.319714 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:41.320060 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:41.320659 (+ 599us) server_negotiation.cc:299] Negotiation successful
0928 16:53:41.321187 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":286,"threads_started":1}
I20260928 16:53:41.323328 24963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:41.417068 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 24840
I20260928 16:53:41.435055 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 24971
I20260928 16:53:41.457614 25153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:41.436455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58452 (local address 127.9.93.3:34841)
0928 16:53:41.437215 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:41.437232 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:41.437286 (+ 54us) server_negotiation.cc:407] Connection header received
0928 16:53:41.439120 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:41.439133 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:41.439805 (+ 672us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:41.440169 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:41.441882 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:41.443484 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:41.445044 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:41.445924 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:41.446140 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:41.449329 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:41.449339 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:41.449360 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:41.449413 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:41.452925 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:41.453927 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:41.453937 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:41.453954 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:41.454183 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:41.454772 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:41.454780 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:41.454782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:41.455720 (+ 938us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:41.456034 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:41.456675 (+ 641us) server_negotiation.cc:299] Negotiation successful
0928 16:53:41.457259 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":371,"threads_started":1}
I20260928 16:53:41.459453 25094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:41.547605 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 24971
2026-09-28T16:53:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25114 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:53:41 dist-test-slave-wzcb krb5kdc[25157](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:53:41 dist-test-slave-wzcb krb5kdc[25157](info): set up 2 sockets
Sep 28 16:53:41 dist-test-slave-wzcb krb5kdc[25157](info): commencing operation
krb5kdc: starting...
W20260928 16:53:42.606487 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:53:42 dist-test-slave-wzcb krb5kdc[25157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:53:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:53:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:53:42.895725 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:43839
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:34633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:43839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:53:43.281322 25170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:43.281682 25170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:43.281807 25170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:43.289789 25170 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 16:53:43.289924 25170 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 16:53:43.289968 25170 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 16:53:43.290043 25170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:53:43.290124 25170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:43.290230 25170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:43.290292 25170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:53:43.290371 25170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:53:43.301388 25170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:34633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:43839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:43839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:43.302944 25170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:43.304620 25170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:43.318192 25175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:43.318225 25176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:43.318550 25170 server_base.cc:1034] running on GCE node
W20260928 16:53:43.318279 25178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:43.319855 25170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:43.321581 25170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:43.322841 25170 hybrid_clock.cc:648] HybridClock initialized: now 1790614423322765 us; error 50 us; skew 500 ppm
I20260928 16:53:43.323226 25170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:43 dist-test-slave-wzcb krb5kdc[25157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614423, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:43.326786 25170 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:53:43.328552 25170 webserver.cc:460] Webserver started at http://127.9.93.62:41865/ using document root <none> and password file <none>
I20260928 16:53:43.329416 25170 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:43.329547 25170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:53:43.329903 25170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:53:43.333478 25170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "a9396a025b064ed1b3cf406e8e7ea609"
format_stamp: "Formatted at 2026-09-28 16:53:43 on dist-test-slave-wzcb"
server_key: "188e7965f23c51a9c935fc6cc1ce8ad8"
server_key_iv: "3b38113c1e2565163687f61457b6f8e1"
server_key_version: "encryptionkey@0"
I20260928 16:53:43.334481 25170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "a9396a025b064ed1b3cf406e8e7ea609"
format_stamp: "Formatted at 2026-09-28 16:53:43 on dist-test-slave-wzcb"
server_key: "188e7965f23c51a9c935fc6cc1ce8ad8"
server_key_iv: "3b38113c1e2565163687f61457b6f8e1"
server_key_version: "encryptionkey@0"
I20260928 16:53:43.341140 25170 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 16:53:43.345768 25185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:43.347702 25170 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:53:43.347930 25170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "a9396a025b064ed1b3cf406e8e7ea609"
format_stamp: "Formatted at 2026-09-28 16:53:43 on dist-test-slave-wzcb"
server_key: "188e7965f23c51a9c935fc6cc1ce8ad8"
server_key_iv: "3b38113c1e2565163687f61457b6f8e1"
server_key_version: "encryptionkey@0"
I20260928 16:53:43.348193 25170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:43.370056 25170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:43.374837 25170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:43.375134 25170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:43.395342 25170 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:43839
I20260928 16:53:43.395334 25236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:43839 every 8 connection(s)
I20260928 16:53:43.397099 25170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:53:43.401698 25237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:53:43.405413 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 25170
I20260928 16:53:43.405635 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:53:43.406376 9588 external_mini_cluster.cc:1428] Setting key 32a4534fd8167b83e31fd646ebe4a0f2
I20260928 16:53:43.416390 25237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609: Bootstrap starting.
I20260928 16:53:43.421097 25237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609: Neither blocks nor log segments found. Creating new log.
I20260928 16:53:43.422781 25237 log.cc:824] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:53:43 dist-test-slave-wzcb krb5kdc[25157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:53:43.426767 25237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609: No bootstrap required, opened a new log
I20260928 16:53:43.433548 25240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:43.408858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36364 (local address 127.9.93.62:43839)
0928 16:53:43.409737 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:43.409752 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:43.409804 (+ 52us) server_negotiation.cc:407] Connection header received
0928 16:53:43.411695 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:43.411725 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:43.412324 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:43.412753 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:43.414803 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:43.416349 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:43.418420 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:43.419246 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:43.419475 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:43.424940 (+ 5465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:43.424949 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:43.424968 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:43.425032 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:43.428405 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:43.429522 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:43.429530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:43.429539 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:43.429771 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:43.430233 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:43.430236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:43.430238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:43.430917 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:43.431198 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:43.431725 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 16:53:43.432231 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":374,"threads_started":1}
I20260928 16:53:43.436474 25237 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9396a025b064ed1b3cf406e8e7ea609" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43839 } }
I20260928 16:53:43.437150 25237 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:53:43.437299 25237 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9396a025b064ed1b3cf406e8e7ea609, State: Initialized, Role: FOLLOWER
I20260928 16:53:43.437934 25237 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9396a025b064ed1b3cf406e8e7ea609" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43839 } }
I20260928 16:53:43.438194 25237 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:53:43.438328 25237 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:53:43.438490 25237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:53:43.441030 25237 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9396a025b064ed1b3cf406e8e7ea609" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43839 } }
I20260928 16:53:43.441653 25237 leader_election.cc:302] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9396a025b064ed1b3cf406e8e7ea609; no voters:
I20260928 16:53:43.442350 25237 leader_election.cc:288] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:53:43.442523 25242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:53:43.443203 25242 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [term 1 LEADER]: Becoming Leader. State: Replica: a9396a025b064ed1b3cf406e8e7ea609, State: Running, Role: LEADER
I20260928 16:53:43.444032 25242 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9396a025b064ed1b3cf406e8e7ea609" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43839 } }
I20260928 16:53:43.445578 25237 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:53:43.446969 25243 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9396a025b064ed1b3cf406e8e7ea609. Latest consensus state: current_term: 1 leader_uuid: "a9396a025b064ed1b3cf406e8e7ea609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9396a025b064ed1b3cf406e8e7ea609" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43839 } } }
I20260928 16:53:43.447499 25243 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [sys.catalog]: This master's current role is: LEADER
I20260928 16:53:43.446961 25244 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9396a025b064ed1b3cf406e8e7ea609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9396a025b064ed1b3cf406e8e7ea609" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43839 } } }
I20260928 16:53:43.448007 25244 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [sys.catalog]: This master's current role is: LEADER
I20260928 16:53:43.449991 25248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:53:43.455828 25248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:53:43.466076 25248 catalog_manager.cc:1347] Generated new cluster ID: 4fe20597e2904c32952bb09c2582b129
I20260928 16:53:43.466223 25248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:53:43.488737 25248 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:53:43.490900 25248 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:53:43.500630 25248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609: Generated new TSK 0
I20260928 16:53:43.501945 25248 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
I20260928 16:53:43.563607 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:43839
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:34633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:53:43.942453 25263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:43.942787 25263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:43.942919 25263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:43.951040 25263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:43.951174 25263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:43.951388 25263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:53:43.962400 25263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:34633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:43839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:43.963973 25263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:43.965499 25263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:43.980377 25268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:43.980410 25271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:43.980433 25269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:43.981487 25263 server_base.cc:1034] running on GCE node
I20260928 16:53:43.982196 25263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:43.983397 25263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:43.984670 25263 hybrid_clock.cc:648] HybridClock initialized: now 1790614423984600 us; error 60 us; skew 500 ppm
I20260928 16:53:43.985069 25263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:43 dist-test-slave-wzcb krb5kdc[25157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614423, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:43.994921 25263 init.cc:377] Logged in from keytab as kudu/127.9.93.1@KRBTEST.COM (short username kudu)
I20260928 16:53:43.996862 25263 webserver.cc:460] Webserver started at http://127.9.93.1:44989/ using document root <none> and password file <none>
I20260928 16:53:43.997740 25263 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:43.997884 25263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:53:43.998252 25263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:53:44.001876 25263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "af87258b04c348719769cc4ffdda32ed"
format_stamp: "Formatted at 2026-09-28 16:53:43 on dist-test-slave-wzcb"
server_key: "f3895f9b0adc933ba606b5fe1906e9ca"
server_key_iv: "401ff682d7c58b51f3545161983434c8"
server_key_version: "encryptionkey@0"
I20260928 16:53:44.002817 25263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "af87258b04c348719769cc4ffdda32ed"
format_stamp: "Formatted at 2026-09-28 16:53:43 on dist-test-slave-wzcb"
server_key: "f3895f9b0adc933ba606b5fe1906e9ca"
server_key_iv: "401ff682d7c58b51f3545161983434c8"
server_key_version: "encryptionkey@0"
I20260928 16:53:44.009441 25263 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 16:53:44.013985 25278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:44.015865 25263 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:53:44.016088 25263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "af87258b04c348719769cc4ffdda32ed"
format_stamp: "Formatted at 2026-09-28 16:53:43 on dist-test-slave-wzcb"
server_key: "f3895f9b0adc933ba606b5fe1906e9ca"
server_key_iv: "401ff682d7c58b51f3545161983434c8"
server_key_version: "encryptionkey@0"
I20260928 16:53:44.016335 25263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:44.032095 25263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:44.036718 25263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:44.037039 25263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:44.038141 25263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:53:44.039710 25263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:53:44.039803 25263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:44.039933 25263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:53:44.040002 25263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:44.073925 25263 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:41141
I20260928 16:53:44.073989 25390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:41141 every 8 connection(s)
I20260928 16:53:44.075390 25263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:53:44.083446 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 25263
I20260928 16:53:44.083710 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:53:44.084637 9588 external_mini_cluster.cc:1428] Setting key d9a375b120f6b9118c2c9fd4332cc3e0
Sep 28 16:53:44 dist-test-slave-wzcb krb5kdc[25157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614423, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:53:44.100112 25393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:44.078367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:47053 (local address 127.9.93.62:43839)
0928 16:53:44.078865 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:44.078871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:44.080114 (+ 1243us) server_negotiation.cc:407] Connection header received
0928 16:53:44.082179 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:44.082185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:44.082287 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:44.082391 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:44.084830 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:44.085760 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:44.088038 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:44.088491 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:44.088603 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:44.093465 (+ 4862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:44.093472 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:44.093476 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:44.093518 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:44.096302 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:44.097423 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:44.097430 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:44.097434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:44.097609 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:44.098058 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:44.098063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:44.098066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:44.098459 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:44.098613 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:44.099503 (+ 890us) server_negotiation.cc:299] Negotiation successful
0928 16:53:44.099760 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":244,"threads_started":1}
I20260928 16:53:44.101156 25394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:44.079013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.1:47053)
0928 16:53:44.079901 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:44.079953 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:44.081835 (+ 1882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:44.082698 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:44.082708 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:44.083546 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:53:44.084549 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:44.084581 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:44.085949 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:44.085953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:44.087842 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:44.087863 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:44.088763 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:44.088770 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:44.089114 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:44.089938 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:53:44.089970 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:53:44.093085 (+ 3115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:53:44.096501 (+ 3416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:44.096512 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:44.096536 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:44.097228 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:44.097713 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:44.097716 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:44.097719 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:44.097938 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:44.098600 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:53:44.098607 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:53:44.099029 (+ 422us) client_negotiation.cc:769] Sending connection context
0928 16:53:44.099332 (+ 303us) client_negotiation.cc:240] Negotiation successful
0928 16:53:44.099844 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":398,"threads_started":1}
I20260928 16:53:44.102946 25391 heartbeater.cc:347] Connected to a master server at 127.9.93.62:43839
I20260928 16:53:44.103325 25391 heartbeater.cc:464] Registering TS with master...
I20260928 16:53:44.104188 25391 heartbeater.cc:511] Master 127.9.93.62:43839 requested a full tablet report, sending...
I20260928 16:53:44.106657 25202 ts_manager.cc:194] Registered new tserver with Master: af87258b04c348719769cc4ffdda32ed (127.9.93.1:41141)
I20260928 16:53:44.108639 25202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.1@KRBTEST.COM'} at 127.9.93.1:47053
WARNING: no policy specified for kudu/127.9.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
I20260928 16:53:44.147342 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:43839
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:34633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:53:44.500319 25397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:44.500675 25397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:44.500829 25397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:44.508548 25397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:44.508719 25397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:44.508889 25397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:53:44.519809 25397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:34633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:43839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:44.521391 25397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:44.522926 25397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:44.537703 25403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:44.537621 25405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:44.537669 25402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:44.538619 25397 server_base.cc:1034] running on GCE node
I20260928 16:53:44.539284 25397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:44.540506 25397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:44.541738 25397 hybrid_clock.cc:648] HybridClock initialized: now 1790614424541680 us; error 47 us; skew 500 ppm
I20260928 16:53:44.542116 25397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:44 dist-test-slave-wzcb krb5kdc[25157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614424, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:44.545898 25397 init.cc:377] Logged in from keytab as kudu/127.9.93.2@KRBTEST.COM (short username kudu)
I20260928 16:53:44.547871 25397 webserver.cc:460] Webserver started at http://127.9.93.2:40607/ using document root <none> and password file <none>
I20260928 16:53:44.548733 25397 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:44.548882 25397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:53:44.549263 25397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:53:44.552948 25397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "0f9d1fc4e00c4e339c566a7e501208d2"
format_stamp: "Formatted at 2026-09-28 16:53:44 on dist-test-slave-wzcb"
server_key: "517dc409e71c298924582b8067522c8e"
server_key_iv: "6cda13483bad7cd8696246a21a854cbd"
server_key_version: "encryptionkey@0"
I20260928 16:53:44.553999 25397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "0f9d1fc4e00c4e339c566a7e501208d2"
format_stamp: "Formatted at 2026-09-28 16:53:44 on dist-test-slave-wzcb"
server_key: "517dc409e71c298924582b8067522c8e"
server_key_iv: "6cda13483bad7cd8696246a21a854cbd"
server_key_version: "encryptionkey@0"
I20260928 16:53:44.560647 25397 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 16:53:44.565285 25412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:44.567066 25397 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 16:53:44.567282 25397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "0f9d1fc4e00c4e339c566a7e501208d2"
format_stamp: "Formatted at 2026-09-28 16:53:44 on dist-test-slave-wzcb"
server_key: "517dc409e71c298924582b8067522c8e"
server_key_iv: "6cda13483bad7cd8696246a21a854cbd"
server_key_version: "encryptionkey@0"
I20260928 16:53:44.567631 25397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:44.591156 25397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:44.595990 25397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:44.596292 25397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:44.597467 25397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:53:44.599048 25397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:53:44.599134 25397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:44.599244 25397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:53:44.599320 25397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:44.632419 25397 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:36357
I20260928 16:53:44.632522 25524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:36357 every 8 connection(s)
I20260928 16:53:44.633864 25397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:53:44.635926 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 25397
I20260928 16:53:44.636142 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:53:44.636708 9588 external_mini_cluster.cc:1428] Setting key 7b57ee23cd3603a30e7201aa4d7806a4
Sep 28 16:53:44 dist-test-slave-wzcb krb5kdc[25157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614424, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:53:44.661089 25527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:44.637158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:58493 (local address 127.9.93.62:43839)
0928 16:53:44.637732 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:44.637738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:44.639081 (+ 1343us) server_negotiation.cc:407] Connection header received
0928 16:53:44.642023 (+ 2942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:44.642028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:44.642124 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:44.642234 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:44.644704 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:44.645554 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:44.648206 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:44.648754 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:44.648883 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:44.653738 (+ 4855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:44.653746 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:44.653750 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:44.653800 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:44.656807 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:44.658110 (+ 1303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:44.658116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:44.658118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:44.658307 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:44.658937 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:44.658943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:44.658947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:44.659293 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:44.659498 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:44.660540 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0928 16:53:44.660772 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":289,"threads_started":1}
I20260928 16:53:44.662688 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:44.637884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.2:58493)
0928 16:53:44.638897 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:44.638974 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:44.641554 (+ 2580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:44.642593 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:44.642607 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:44.643503 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:53:44.644452 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:44.644481 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:44.645738 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:44.645744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:44.647999 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:44.648019 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:44.649030 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:44.649037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:44.649358 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:44.650269 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:53:44.650301 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:53:44.653377 (+ 3076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:53:44.657045 (+ 3668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:44.657056 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:44.657081 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:44.657894 (+ 813us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:44.658458 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:44.658464 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:44.658467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:44.658779 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:44.659485 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:53:44.659494 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:53:44.660070 (+ 576us) client_negotiation.cc:769] Sending connection context
0928 16:53:44.660438 (+ 368us) client_negotiation.cc:240] Negotiation successful
0928 16:53:44.661056 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":367,"threads_started":1}
I20260928 16:53:44.664731 25525 heartbeater.cc:347] Connected to a master server at 127.9.93.62:43839
I20260928 16:53:44.665119 25525 heartbeater.cc:464] Registering TS with master...
I20260928 16:53:44.665964 25525 heartbeater.cc:511] Master 127.9.93.62:43839 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.93.3@KRBTEST.COM; defaulting to no policy
I20260928 16:53:44.667912 25202 ts_manager.cc:194] Registered new tserver with Master: 0f9d1fc4e00c4e339c566a7e501208d2 (127.9.93.2:36357)
I20260928 16:53:44.668824 25202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.2@KRBTEST.COM'} at 127.9.93.2:58493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
I20260928 16:53:44.702943 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:43839
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:34633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:53:45.063750 25531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:53:45.064088 25531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:53:45.064209 25531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:53:45.071892 25531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:53:45.072064 25531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:53:45.072203 25531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:53:45.082692 25531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:34633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:43839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:53:45.084197 25531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:53:45.085669 25531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:53:45.100678 25537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:45.100598 25539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:53:45.100631 25536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:53:45.109463 25531 server_base.cc:1034] running on GCE node
I20260928 16:53:45.110378 25531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:53:45.111676 25531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:53:45.112202 25391 heartbeater.cc:503] Master 127.9.93.62:43839 was elected leader, sending a full tablet report...
I20260928 16:53:45.112993 25531 hybrid_clock.cc:648] HybridClock initialized: now 1790614425112907 us; error 69 us; skew 500 ppm
I20260928 16:53:45.113382 25531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:53:45 dist-test-slave-wzcb krb5kdc[25157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614425, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:53:45.117079 25531 init.cc:377] Logged in from keytab as kudu/127.9.93.3@KRBTEST.COM (short username kudu)
I20260928 16:53:45.118824 25531 webserver.cc:460] Webserver started at http://127.9.93.3:41393/ using document root <none> and password file <none>
I20260928 16:53:45.119731 25531 fs_manager.cc:359] Metadata directory not provided
I20260928 16:53:45.119895 25531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:53:45.120306 25531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:53:45.123858 25531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "af8fe338838d437ca36bcaf2fc0d89b6"
format_stamp: "Formatted at 2026-09-28 16:53:45 on dist-test-slave-wzcb"
server_key: "ec918ffc3df798e2f3554c756ca54273"
server_key_iv: "bb55c5fd2dc5acf93f75b56a63360d48"
server_key_version: "encryptionkey@0"
I20260928 16:53:45.124775 25531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "af8fe338838d437ca36bcaf2fc0d89b6"
format_stamp: "Formatted at 2026-09-28 16:53:45 on dist-test-slave-wzcb"
server_key: "ec918ffc3df798e2f3554c756ca54273"
server_key_iv: "bb55c5fd2dc5acf93f75b56a63360d48"
server_key_version: "encryptionkey@0"
I20260928 16:53:45.131138 25531 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 16:53:45.135612 25546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:45.137303 25531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:53:45.137516 25531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "af8fe338838d437ca36bcaf2fc0d89b6"
format_stamp: "Formatted at 2026-09-28 16:53:45 on dist-test-slave-wzcb"
server_key: "ec918ffc3df798e2f3554c756ca54273"
server_key_iv: "bb55c5fd2dc5acf93f75b56a63360d48"
server_key_version: "encryptionkey@0"
I20260928 16:53:45.137761 25531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:53:45.153290 25531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:53:45.158227 25531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:53:45.158562 25531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:53:45.159792 25531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:53:45.161413 25531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:53:45.161505 25531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:45.161638 25531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:53:45.161698 25531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:53:45.194131 25531 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:36555
I20260928 16:53:45.194180 25658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:36555 every 8 connection(s)
I20260928 16:53:45.195571 25531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:53:45.202198 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 25531
I20260928 16:53:45.202423 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:53:45.203091 9588 external_mini_cluster.cc:1428] Setting key c6bba5d617ddb2c8d97f665f468f6859
Sep 28 16:53:45 dist-test-slave-wzcb krb5kdc[25157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614425, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:53:45.221577 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:45.198896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:50281 (local address 127.9.93.62:43839)
0928 16:53:45.199557 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:45.199565 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:45.201086 (+ 1521us) server_negotiation.cc:407] Connection header received
0928 16:53:45.204133 (+ 3047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:45.204140 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:45.204270 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:45.204409 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:45.207131 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.208029 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.209826 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.210344 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.210462 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:45.214383 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:45.214389 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:45.214393 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:45.214436 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:45.217509 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:45.218568 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:45.218575 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:45.218578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:45.218749 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:45.219292 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:45.219297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:45.219317 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:45.219709 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:45.219883 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:45.220905 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0928 16:53:45.221195 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":279,"threads_started":1}
I20260928 16:53:45.222615 25662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:45.199696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.3:50281)
0928 16:53:45.200857 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:45.200915 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:45.203634 (+ 2719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:45.204880 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:45.204896 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:45.205826 (+ 930us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:53:45.206861 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:45.206894 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.208202 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.208207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:45.209639 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:45.209654 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.210615 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.210621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:45.210850 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:45.211638 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:53:45.211664 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:53:45.214088 (+ 2424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:53:45.217765 (+ 3677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:45.217784 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:45.217806 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:45.218438 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:45.218884 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:53:45.218889 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:53:45.218893 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:53:45.219167 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:53:45.219860 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:53:45.219870 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:53:45.220470 (+ 600us) client_negotiation.cc:769] Sending connection context
0928 16:53:45.220794 (+ 324us) client_negotiation.cc:240] Negotiation successful
0928 16:53:45.221299 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"thread_start_us":573,"threads_started":1}
I20260928 16:53:45.224210 25659 heartbeater.cc:347] Connected to a master server at 127.9.93.62:43839
I20260928 16:53:45.224565 25659 heartbeater.cc:464] Registering TS with master...
I20260928 16:53:45.225375 25659 heartbeater.cc:511] Master 127.9.93.62:43839 requested a full tablet report, sending...
I20260928 16:53:45.227098 25202 ts_manager.cc:194] Registered new tserver with Master: af8fe338838d437ca36bcaf2fc0d89b6 (127.9.93.3:36555)
I20260928 16:53:45.228008 25202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.3@KRBTEST.COM'} at 127.9.93.3:50281
I20260928 16:53:45.233256 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:53:45.252693 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:45.238190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36380 (local address 127.9.93.62:43839)
0928 16:53:45.238443 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:45.238452 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:45.238859 (+ 407us) server_negotiation.cc:407] Connection header received
0928 16:53:45.238988 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:45.238992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:45.239067 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:45.239152 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:45.240766 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.241553 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.243062 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.243624 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.243743 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:45.246401 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:45.246407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:45.246410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:45.246454 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:45.249475 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:45.250591 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:45.250596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:45.250599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:45.250764 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:45.251191 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:45.251198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:45.251201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:45.251649 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:45.251858 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:45.252159 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 16:53:45.252394 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 16:53:45.257263 25202 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:53:45.264595 25202 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:53:45.307596 25674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:45.279526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:36357 (local address 127.0.0.1:47094)
0928 16:53:45.297443 (+ 17917us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:45.297517 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:45.297717 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:45.298996 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:45.299003 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:45.299036 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:53:45.299538 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:45.299565 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.301500 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.301505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:45.304068 (+ 2563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:45.304083 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.306437 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.306443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:45.306662 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:45.306718 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 16:53:45.306853 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 16:53:45.307111 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17590,"mutex_wait_us":46,"thread_start_us":18878,"threads_started":1}
I20260928 16:53:45.308606 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:45.279703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47094 (local address 127.9.93.2:36357)
0928 16:53:45.295896 (+ 16193us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:45.295903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:45.297957 (+ 2054us) server_negotiation.cc:407] Connection header received
0928 16:53:45.298081 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:45.298090 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:45.298435 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:45.298660 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:53:45.299744 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.301337 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.304283 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.305887 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.306027 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:45.306275 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:45.308057 (+ 1782us) server_negotiation.cc:299] Negotiation successful
0928 16:53:45.308264 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16010,"thread_start_us":15848,"threads_started":1}
I20260928 16:53:45.309005 25672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:45.280008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:36555 (local address 127.0.0.1:58078)
0928 16:53:45.298713 (+ 18705us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:45.298733 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:45.298864 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:45.299994 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:45.299998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:45.300024 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:53:45.300396 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:45.300410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.302363 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.302368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:45.306145 (+ 3777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:45.306162 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.308224 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.308231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:45.308418 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:45.308453 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:53:45.308520 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 16:53:45.308658 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18574,"thread_start_us":16965,"threads_started":1}
I20260928 16:53:45.310073 25675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:45.280392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58078 (local address 127.9.93.3:36555)
0928 16:53:45.297086 (+ 16694us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:45.297097 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:45.299003 (+ 1906us) server_negotiation.cc:407] Connection header received
0928 16:53:45.299117 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:45.299125 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:45.299449 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:45.299663 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:53:45.300631 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.302186 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.306384 (+ 4198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.307983 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.309150 (+ 1167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:45.309423 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:45.309503 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 16:53:45.309704 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16479,"thread_start_us":16168,"threads_started":1}
I20260928 16:53:45.314764 25594 tablet_service.cc:1467] Processing CreateTablet for tablet c803cba31a18489480795263c842b97a (DEFAULT_TABLE table=test-table [id=1494ffe962b04e22befc586da5eaff24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:53:45.314611 25460 tablet_service.cc:1467] Processing CreateTablet for tablet c803cba31a18489480795263c842b97a (DEFAULT_TABLE table=test-table [id=1494ffe962b04e22befc586da5eaff24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:53:45.316424 25594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c803cba31a18489480795263c842b97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:53:45.316437 25460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c803cba31a18489480795263c842b97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:53:45.318094 25674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:45.298798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:41141 (local address 127.0.0.1:58730)
0928 16:53:45.309336 (+ 10538us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:45.309357 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:45.309477 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:45.312593 (+ 3116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:45.312598 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:45.312628 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:53:45.312823 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:45.312837 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.314168 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.314172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:45.315441 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:45.315460 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.317229 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.317235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:45.317424 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:45.317491 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 16:53:45.317585 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 16:53:45.317762 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10419,"thread_start_us":11630,"threads_started":1}
I20260928 16:53:45.318140 25676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:45.298683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.9.93.1:41141)
0928 16:53:45.311145 (+ 12462us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:45.311155 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:45.311189 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:53:45.311466 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:45.311476 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:45.312391 (+ 915us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:45.312593 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:53:45.312987 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.314017 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.315643 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.317053 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.317181 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:45.317459 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:45.317647 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 16:53:45.317836 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11916,"thread_start_us":11756,"threads_started":1}
I20260928 16:53:45.322471 25326 tablet_service.cc:1467] Processing CreateTablet for tablet c803cba31a18489480795263c842b97a (DEFAULT_TABLE table=test-table [id=1494ffe962b04e22befc586da5eaff24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:53:45.323954 25326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c803cba31a18489480795263c842b97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:53:45.333817 25678 tablet_bootstrap.cc:492] T c803cba31a18489480795263c842b97a P af8fe338838d437ca36bcaf2fc0d89b6: Bootstrap starting.
I20260928 16:53:45.337857 25680 tablet_bootstrap.cc:492] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed: Bootstrap starting.
I20260928 16:53:45.338747 25678 tablet_bootstrap.cc:654] T c803cba31a18489480795263c842b97a P af8fe338838d437ca36bcaf2fc0d89b6: Neither blocks nor log segments found. Creating new log.
I20260928 16:53:45.339099 25679 tablet_bootstrap.cc:492] T c803cba31a18489480795263c842b97a P 0f9d1fc4e00c4e339c566a7e501208d2: Bootstrap starting.
I20260928 16:53:45.340242 25678 log.cc:824] T c803cba31a18489480795263c842b97a P af8fe338838d437ca36bcaf2fc0d89b6: Log is configured to *not* fsync() on all Append() calls
I20260928 16:53:45.342828 25679 tablet_bootstrap.cc:654] T c803cba31a18489480795263c842b97a P 0f9d1fc4e00c4e339c566a7e501208d2: Neither blocks nor log segments found. Creating new log.
I20260928 16:53:45.342839 25680 tablet_bootstrap.cc:654] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed: Neither blocks nor log segments found. Creating new log.
I20260928 16:53:45.343552 25678 tablet_bootstrap.cc:492] T c803cba31a18489480795263c842b97a P af8fe338838d437ca36bcaf2fc0d89b6: No bootstrap required, opened a new log
I20260928 16:53:45.343843 25678 ts_tablet_manager.cc:1397] T c803cba31a18489480795263c842b97a P af8fe338838d437ca36bcaf2fc0d89b6: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260928 16:53:45.344310 25680 log.cc:824] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed: Log is configured to *not* fsync() on all Append() calls
I20260928 16:53:45.344310 25679 log.cc:824] T c803cba31a18489480795263c842b97a P 0f9d1fc4e00c4e339c566a7e501208d2: Log is configured to *not* fsync() on all Append() calls
I20260928 16:53:45.353206 25678 raft_consensus.cc:348] T c803cba31a18489480795263c842b97a P af8fe338838d437ca36bcaf2fc0d89b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9d1fc4e00c4e339c566a7e501208d2" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 36357 } } peers { permanent_uuid: "af8fe338838d437ca36bcaf2fc0d89b6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36555 } } peers { permanent_uuid: "af87258b04c348719769cc4ffdda32ed" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 41141 } }
I20260928 16:53:45.353675 25678 raft_consensus.cc:374] T c803cba31a18489480795263c842b97a P af8fe338838d437ca36bcaf2fc0d89b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:53:45.353776 25678 raft_consensus.cc:729] T c803cba31a18489480795263c842b97a P af8fe338838d437ca36bcaf2fc0d89b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af8fe338838d437ca36bcaf2fc0d89b6, State: Initialized, Role: FOLLOWER
I20260928 16:53:45.354565 25680 tablet_bootstrap.cc:492] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed: No bootstrap required, opened a new log
I20260928 16:53:45.354552 25679 tablet_bootstrap.cc:492] T c803cba31a18489480795263c842b97a P 0f9d1fc4e00c4e339c566a7e501208d2: No bootstrap required, opened a new log
I20260928 16:53:45.354456 25678 consensus_queue.cc:260] T c803cba31a18489480795263c842b97a P af8fe338838d437ca36bcaf2fc0d89b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9d1fc4e00c4e339c566a7e501208d2" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 36357 } } peers { permanent_uuid: "af8fe338838d437ca36bcaf2fc0d89b6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36555 } } peers { permanent_uuid: "af87258b04c348719769cc4ffdda32ed" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 41141 } }
I20260928 16:53:45.354905 25679 ts_tablet_manager.cc:1397] T c803cba31a18489480795263c842b97a P 0f9d1fc4e00c4e339c566a7e501208d2: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.008s
I20260928 16:53:45.354905 25680 ts_tablet_manager.cc:1397] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20260928 16:53:45.355721 25659 heartbeater.cc:503] Master 127.9.93.62:43839 was elected leader, sending a full tablet report...
I20260928 16:53:45.356492 25678 ts_tablet_manager.cc:1428] T c803cba31a18489480795263c842b97a P af8fe338838d437ca36bcaf2fc0d89b6: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260928 16:53:45.366221 25679 raft_consensus.cc:348] T c803cba31a18489480795263c842b97a P 0f9d1fc4e00c4e339c566a7e501208d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9d1fc4e00c4e339c566a7e501208d2" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 36357 } } peers { permanent_uuid: "af8fe338838d437ca36bcaf2fc0d89b6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36555 } } peers { permanent_uuid: "af87258b04c348719769cc4ffdda32ed" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 41141 } }
I20260928 16:53:45.366221 25680 raft_consensus.cc:348] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9d1fc4e00c4e339c566a7e501208d2" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 36357 } } peers { permanent_uuid: "af8fe338838d437ca36bcaf2fc0d89b6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36555 } } peers { permanent_uuid: "af87258b04c348719769cc4ffdda32ed" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 41141 } }
I20260928 16:53:45.366799 25680 raft_consensus.cc:374] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:53:45.366799 25679 raft_consensus.cc:374] T c803cba31a18489480795263c842b97a P 0f9d1fc4e00c4e339c566a7e501208d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:53:45.366920 25679 raft_consensus.cc:729] T c803cba31a18489480795263c842b97a P 0f9d1fc4e00c4e339c566a7e501208d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f9d1fc4e00c4e339c566a7e501208d2, State: Initialized, Role: FOLLOWER
I20260928 16:53:45.366925 25680 raft_consensus.cc:729] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af87258b04c348719769cc4ffdda32ed, State: Initialized, Role: FOLLOWER
I20260928 16:53:45.367749 25680 consensus_queue.cc:260] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9d1fc4e00c4e339c566a7e501208d2" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 36357 } } peers { permanent_uuid: "af8fe338838d437ca36bcaf2fc0d89b6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36555 } } peers { permanent_uuid: "af87258b04c348719769cc4ffdda32ed" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 41141 } }
I20260928 16:53:45.367749 25679 consensus_queue.cc:260] T c803cba31a18489480795263c842b97a P 0f9d1fc4e00c4e339c566a7e501208d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9d1fc4e00c4e339c566a7e501208d2" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 36357 } } peers { permanent_uuid: "af8fe338838d437ca36bcaf2fc0d89b6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36555 } } peers { permanent_uuid: "af87258b04c348719769cc4ffdda32ed" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 41141 } }
I20260928 16:53:45.369065 25525 heartbeater.cc:503] Master 127.9.93.62:43839 was elected leader, sending a full tablet report...
I20260928 16:53:45.370050 25679 ts_tablet_manager.cc:1428] T c803cba31a18489480795263c842b97a P 0f9d1fc4e00c4e339c566a7e501208d2: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260928 16:53:45.370236 25680 ts_tablet_manager.cc:1428] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed: Time spent starting tablet: real 0.015s user 0.016s sys 0.001s
W20260928 16:53:45.386032 25526 tablet.cc:2369] T c803cba31a18489480795263c842b97a P 0f9d1fc4e00c4e339c566a7e501208d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:53:45.447440 25660 tablet.cc:2369] T c803cba31a18489480795263c842b97a P af8fe338838d437ca36bcaf2fc0d89b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:53:45.459486 25686 raft_consensus.cc:482] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:53:45.459815 25686 raft_consensus.cc:504] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9d1fc4e00c4e339c566a7e501208d2" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 36357 } } peers { permanent_uuid: "af8fe338838d437ca36bcaf2fc0d89b6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36555 } } peers { permanent_uuid: "af87258b04c348719769cc4ffdda32ed" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 41141 } }
I20260928 16:53:45.461701 25686 leader_election.cc:288] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f9d1fc4e00c4e339c566a7e501208d2 (127.9.93.2:36357), af8fe338838d437ca36bcaf2fc0d89b6 (127.9.93.3:36555)
I20260928 16:53:45.468672 25675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:45.462189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:35487 (local address 127.9.93.3:36555)
0928 16:53:45.462413 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:45.462420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:45.463071 (+ 651us) server_negotiation.cc:407] Connection header received
0928 16:53:45.463226 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:45.463230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:45.463295 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:45.463409 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:53:45.464104 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.464858 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.466825 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.467712 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.467830 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:45.467949 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:45.468282 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 16:53:45.468412 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 16:53:45.468693 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:45.462228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:60017 (local address 127.9.93.2:36357)
0928 16:53:45.462440 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:45.462448 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:45.462945 (+ 497us) server_negotiation.cc:407] Connection header received
0928 16:53:45.463220 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:45.463225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:45.463294 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:45.463410 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:53:45.464138 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.464838 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.466827 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.467717 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.467848 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:45.467957 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:45.468317 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 16:53:45.468445 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:53:45.468851 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:45.462199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:36555 (local address 127.9.93.1:35487)
0928 16:53:45.462955 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:45.462970 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:45.463092 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:45.463440 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:45.463444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:45.463469 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:53:45.463918 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:45.463928 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.464995 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.464999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:45.466666 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:45.466679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.467957 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.467964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:45.468117 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:45.468147 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 16:53:45.468224 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 16:53:45.468369 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":28,"spinlock_wait_cycles":35712,"thread_start_us":444,"threads_started":1}
I20260928 16:53:45.469106 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:45.461980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:36357 (local address 127.9.93.1:60017)
0928 16:53:45.462814 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0928 16:53:45.462831 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:53:45.463018 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:53:45.463441 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:53:45.463445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:53:45.463549 (+ 104us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:53:45.463986 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:45.463995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.464985 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.464991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:45.466650 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:53:45.466664 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.468021 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.468026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:53:45.468155 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:45.468178 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:53:45.468268 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 16:53:45.468407 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":23,"spinlock_wait_cycles":29952,"thread_start_us":471,"threads_started":1}
I20260928 16:53:45.470005 25614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c803cba31a18489480795263c842b97a" candidate_uuid: "af87258b04c348719769cc4ffdda32ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8fe338838d437ca36bcaf2fc0d89b6" is_pre_election: true
I20260928 16:53:45.470065 25480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c803cba31a18489480795263c842b97a" candidate_uuid: "af87258b04c348719769cc4ffdda32ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f9d1fc4e00c4e339c566a7e501208d2" is_pre_election: true
I20260928 16:53:45.470533 25480 raft_consensus.cc:2454] T c803cba31a18489480795263c842b97a P 0f9d1fc4e00c4e339c566a7e501208d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af87258b04c348719769cc4ffdda32ed in term 0.
I20260928 16:53:45.470530 25614 raft_consensus.cc:2454] T c803cba31a18489480795263c842b97a P af8fe338838d437ca36bcaf2fc0d89b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af87258b04c348719769cc4ffdda32ed in term 0.
I20260928 16:53:45.471239 25280 leader_election.cc:302] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9d1fc4e00c4e339c566a7e501208d2, af87258b04c348719769cc4ffdda32ed; no voters:
I20260928 16:53:45.471709 25686 raft_consensus.cc:2781] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:53:45.471859 25686 raft_consensus.cc:482] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:53:45.471954 25686 raft_consensus.cc:3037] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:53:45.474475 25686 raft_consensus.cc:504] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9d1fc4e00c4e339c566a7e501208d2" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 36357 } } peers { permanent_uuid: "af8fe338838d437ca36bcaf2fc0d89b6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36555 } } peers { permanent_uuid: "af87258b04c348719769cc4ffdda32ed" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 41141 } }
I20260928 16:53:45.475158 25686 leader_election.cc:288] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed [CANDIDATE]: Term 1 election: Requested vote from peers 0f9d1fc4e00c4e339c566a7e501208d2 (127.9.93.2:36357), af8fe338838d437ca36bcaf2fc0d89b6 (127.9.93.3:36555)
I20260928 16:53:45.475674 25480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c803cba31a18489480795263c842b97a" candidate_uuid: "af87258b04c348719769cc4ffdda32ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f9d1fc4e00c4e339c566a7e501208d2"
I20260928 16:53:45.475718 25614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c803cba31a18489480795263c842b97a" candidate_uuid: "af87258b04c348719769cc4ffdda32ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8fe338838d437ca36bcaf2fc0d89b6"
I20260928 16:53:45.475921 25480 raft_consensus.cc:3037] T c803cba31a18489480795263c842b97a P 0f9d1fc4e00c4e339c566a7e501208d2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:53:45.475934 25614 raft_consensus.cc:3037] T c803cba31a18489480795263c842b97a P af8fe338838d437ca36bcaf2fc0d89b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:53:45.479228 25480 raft_consensus.cc:2454] T c803cba31a18489480795263c842b97a P 0f9d1fc4e00c4e339c566a7e501208d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af87258b04c348719769cc4ffdda32ed in term 1.
I20260928 16:53:45.479229 25614 raft_consensus.cc:2454] T c803cba31a18489480795263c842b97a P af8fe338838d437ca36bcaf2fc0d89b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af87258b04c348719769cc4ffdda32ed in term 1.
I20260928 16:53:45.479765 25282 leader_election.cc:302] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af87258b04c348719769cc4ffdda32ed, af8fe338838d437ca36bcaf2fc0d89b6; no voters:
I20260928 16:53:45.480134 25686 raft_consensus.cc:2781] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:53:45.480644 25686 raft_consensus.cc:686] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed [term 1 LEADER]: Becoming Leader. State: Replica: af87258b04c348719769cc4ffdda32ed, State: Running, Role: LEADER
I20260928 16:53:45.481171 25686 consensus_queue.cc:237] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9d1fc4e00c4e339c566a7e501208d2" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 36357 } } peers { permanent_uuid: "af8fe338838d437ca36bcaf2fc0d89b6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36555 } } peers { permanent_uuid: "af87258b04c348719769cc4ffdda32ed" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 41141 } }
I20260928 16:53:45.486428 25202 catalog_manager.cc:5495] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed reported cstate change: term changed from 0 to 1, leader changed from <none> to af87258b04c348719769cc4ffdda32ed (127.9.93.1). New cstate: current_term: 1 leader_uuid: "af87258b04c348719769cc4ffdda32ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9d1fc4e00c4e339c566a7e501208d2" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 36357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af8fe338838d437ca36bcaf2fc0d89b6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af87258b04c348719769cc4ffdda32ed" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 41141 } health_report { overall_health: HEALTHY } } }
I20260928 16:53:45.534577 25676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:45.528654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58742 (local address 127.9.93.1:41141)
0928 16:53:45.528895 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:45.528899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:45.528920 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:53:45.529102 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:45.529106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:45.529215 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:45.529350 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:53:45.529737 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.530464 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.532170 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.532615 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.532721 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:45.533310 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:53:45.533997 (+ 687us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:53:45.534119 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:45.534205 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 16:53:45.534287 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 16:53:45.544289 25614 raft_consensus.cc:1261] T c803cba31a18489480795263c842b97a P af8fe338838d437ca36bcaf2fc0d89b6 [term 1 FOLLOWER]: Refusing update from remote peer af87258b04c348719769cc4ffdda32ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:53:45.544296 25480 raft_consensus.cc:1261] T c803cba31a18489480795263c842b97a P 0f9d1fc4e00c4e339c566a7e501208d2 [term 1 FOLLOWER]: Refusing update from remote peer af87258b04c348719769cc4ffdda32ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:53:45.545404 25686 consensus_queue.cc:1035] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "af8fe338838d437ca36bcaf2fc0d89b6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:53:45.545790 25689 consensus_queue.cc:1035] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f9d1fc4e00c4e339c566a7e501208d2" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 36357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:53:45.556388 25692 mvcc.cc:205] Tried to move back new op lower bound from 7334356687017709568 to 7334356686775070720. Current Snapshot: MvccSnapshot[applied={T|T < 7334356687017709568}]
I20260928 16:53:45.559465 25693 mvcc.cc:205] Tried to move back new op lower bound from 7334356687017709568 to 7334356686775070720. Current Snapshot: MvccSnapshot[applied={T|T < 7334356687017709568}]
I20260928 16:53:45.577755 25202 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:53:45.578059 25202 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:53:45.582216 25202 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609: Sending DeleteTablet for 3 replicas of tablet c803cba31a18489480795263c842b97a
I20260928 16:53:45.583638 25460 tablet_service.cc:1514] Processing DeleteTablet for tablet c803cba31a18489480795263c842b97a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:53:45 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:47094
I20260928 16:53:45.583843 25594 tablet_service.cc:1514] Processing DeleteTablet for tablet c803cba31a18489480795263c842b97a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:53:45 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:58078
I20260928 16:53:45.584054 25326 tablet_service.cc:1514] Processing DeleteTablet for tablet c803cba31a18489480795263c842b97a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:53:45 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:58730
I20260928 16:53:45.584740 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 25263
I20260928 16:53:45.596233 25701 tablet_replica.cc:330] T c803cba31a18489480795263c842b97a P 0f9d1fc4e00c4e339c566a7e501208d2: stopping tablet replica
I20260928 16:53:45.597078 25701 raft_consensus.cc:2229] T c803cba31a18489480795263c842b97a P 0f9d1fc4e00c4e339c566a7e501208d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:53:45.597888 25701 raft_consensus.cc:2258] T c803cba31a18489480795263c842b97a P 0f9d1fc4e00c4e339c566a7e501208d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:53:45.599211 25703 tablet_replica.cc:330] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed: stopping tablet replica
I20260928 16:53:45.599881 25703 raft_consensus.cc:2229] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:53:45.600868 25703 raft_consensus.cc:2258] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:53:45.600860 25701 ts_tablet_manager.cc:1905] T c803cba31a18489480795263c842b97a P 0f9d1fc4e00c4e339c566a7e501208d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:53:45.603601 25703 ts_tablet_manager.cc:1905] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:53:45.608911 25701 ts_tablet_manager.cc:1918] T c803cba31a18489480795263c842b97a P 0f9d1fc4e00c4e339c566a7e501208d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:53:45.609143 25701 log.cc:1196] T c803cba31a18489480795263c842b97a P 0f9d1fc4e00c4e339c566a7e501208d2: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/c803cba31a18489480795263c842b97a
I20260928 16:53:45.609730 25701 ts_tablet_manager.cc:1939] T c803cba31a18489480795263c842b97a P 0f9d1fc4e00c4e339c566a7e501208d2: Deleting consensus metadata
I20260928 16:53:45.612059 25187 catalog_manager.cc:4841] TS 0f9d1fc4e00c4e339c566a7e501208d2 (127.9.93.2:36357): tablet c803cba31a18489480795263c842b97a (table test-table [id=1494ffe962b04e22befc586da5eaff24]) successfully deleted
I20260928 16:53:45.612494 25703 ts_tablet_manager.cc:1918] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:53:45.612860 25703 log.cc:1196] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/wal/wals/c803cba31a18489480795263c842b97a
I20260928 16:53:45.613131 25702 tablet_replica.cc:330] T c803cba31a18489480795263c842b97a P af8fe338838d437ca36bcaf2fc0d89b6: stopping tablet replica
I20260928 16:53:45.613637 25703 ts_tablet_manager.cc:1939] T c803cba31a18489480795263c842b97a P af87258b04c348719769cc4ffdda32ed: Deleting consensus metadata
I20260928 16:53:45.613652 25702 raft_consensus.cc:2229] T c803cba31a18489480795263c842b97a P af8fe338838d437ca36bcaf2fc0d89b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:53:45.614252 25702 raft_consensus.cc:2258] T c803cba31a18489480795263c842b97a P af8fe338838d437ca36bcaf2fc0d89b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:53:45.615989 25189 catalog_manager.cc:4841] TS af87258b04c348719769cc4ffdda32ed (127.9.93.1:41141): tablet c803cba31a18489480795263c842b97a (table test-table [id=1494ffe962b04e22befc586da5eaff24]) successfully deleted
I20260928 16:53:45.616966 25702 ts_tablet_manager.cc:1905] T c803cba31a18489480795263c842b97a P af8fe338838d437ca36bcaf2fc0d89b6: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 16:53:45 dist-test-slave-wzcb krb5kdc[25157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.1@KRBTEST.COM
I20260928 16:53:45.624842 25702 ts_tablet_manager.cc:1918] T c803cba31a18489480795263c842b97a P af8fe338838d437ca36bcaf2fc0d89b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:53:45.625080 25702 log.cc:1196] T c803cba31a18489480795263c842b97a P af8fe338838d437ca36bcaf2fc0d89b6: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/c803cba31a18489480795263c842b97a
I20260928 16:53:45.625662 25702 ts_tablet_manager.cc:1939] T c803cba31a18489480795263c842b97a P af8fe338838d437ca36bcaf2fc0d89b6: Deleting consensus metadata
I20260928 16:53:45.627892 25189 catalog_manager.cc:4841] TS af8fe338838d437ca36bcaf2fc0d89b6 (127.9.93.3:36555): tablet c803cba31a18489480795263c842b97a (table test-table [id=1494ffe962b04e22befc586da5eaff24]) successfully deleted
I20260928 16:53:45.629241 25676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:45.586119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58750 (local address 127.9.93.1:41141)
0928 16:53:45.586375 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:45.586387 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:45.610970 (+ 24583us) server_negotiation.cc:407] Connection header received
0928 16:53:45.611562 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:45.611568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:45.611657 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:45.611731 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:45.614280 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.615558 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.617300 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.617760 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.617887 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:45.622538 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:45.622553 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:45.622569 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:45.622616 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:45.625960 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:45.626806 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:45.626820 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:45.626822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:45.626989 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:45.627434 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:45.627440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:45.627443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:45.627903 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:45.628189 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:45.628617 (+ 428us) server_negotiation.cc:299] Negotiation successful
0928 16:53:45.628877 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 16:53:45.630048 25386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:45.734997 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 25263
I20260928 16:53:45.753697 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 25397
Sep 28 16:53:45 dist-test-slave-wzcb krb5kdc[25157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.2@KRBTEST.COM
I20260928 16:53:45.770576 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:45.754474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47100 (local address 127.9.93.2:36357)
0928 16:53:45.754689 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:45.754694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:45.754713 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:53:45.754872 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:45.754876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:45.754961 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:45.755060 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:45.756530 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.757301 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.758776 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.759185 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.759279 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:45.764063 (+ 4784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:45.764083 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:45.764102 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:45.764149 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:45.767215 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:45.768183 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:45.768193 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:45.768195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:45.768357 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:45.768798 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:45.768801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:45.768804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:45.769259 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:45.769580 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:45.769944 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 16:53:45.770199 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260928 16:53:45.771277 25520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:45.863464 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 25397
I20260928 16:53:45.878841 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 25531
Sep 28 16:53:45 dist-test-slave-wzcb krb5kdc[25157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.3@KRBTEST.COM
I20260928 16:53:45.896325 25675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:53:45.879747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58082 (local address 127.9.93.3:36555)
0928 16:53:45.879952 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 16:53:45.879957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:53:45.879978 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:53:45.880208 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:53:45.880213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:53:45.880293 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:53:45.880409 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:53:45.881999 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.882784 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.884432 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:53:45.884893 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:53:45.885016 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:53:45.889920 (+ 4904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:53:45.889940 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:53:45.889958 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:53:45.890008 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:53:45.893133 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:45.894093 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:45.894105 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:45.894107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:45.894273 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:53:45.894719 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:53:45.894722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:53:45.894724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:53:45.895135 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:53:45.895446 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0928 16:53:45.895814 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 16:53:45.896027 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 16:53:45.897053 25654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:45.987738 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 25531
I20260928 16:53:46.002519 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 25170
I20260928 16:53:46.003871 25232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:53:46.121600 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 25170
I20260928 16:54:06.145030 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:43839
--webserver_interface=127.9.93.62
--webserver_port=41865
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:34633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:43839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:54:06.533236 25709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:54:06.533593 25709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:54:06.533713 25709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:54:06.541695 25709 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 16:54:06.541826 25709 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 16:54:06.541865 25709 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 16:54:06.541929 25709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:54:06.542012 25709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:54:06.542111 25709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:54:06.542172 25709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:54:06.542224 25709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:54:06.553165 25709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:34633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:43839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:43839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=41865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:54:06.554713 25709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:54:06.556330 25709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:54:06.585074 25715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:06.585045 25716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:06.585099 25718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:54:06.585933 25709 server_base.cc:1034] running on GCE node
I20260928 16:54:06.587160 25709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:54:06.589283 25709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:54:06.590605 25709 hybrid_clock.cc:648] HybridClock initialized: now 1790614446590513 us; error 70 us; skew 500 ppm
I20260928 16:54:06.591100 25709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:54:06 dist-test-slave-wzcb krb5kdc[25157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614446, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:54:06.600154 25709 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:54:06.602331 25709 webserver.cc:460] Webserver started at http://127.9.93.62:41865/ using document root <none> and password file <none>
I20260928 16:54:06.603394 25709 fs_manager.cc:359] Metadata directory not provided
I20260928 16:54:06.603570 25709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:54:06.611410 25709 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.002s
I20260928 16:54:06.615523 25725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:06.617460 25709 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260928 16:54:06.617733 25709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "a9396a025b064ed1b3cf406e8e7ea609"
format_stamp: "Formatted at 2026-09-28 16:53:43 on dist-test-slave-wzcb"
server_key: "188e7965f23c51a9c935fc6cc1ce8ad8"
server_key_iv: "3b38113c1e2565163687f61457b6f8e1"
server_key_version: "encryptionkey@0"
I20260928 16:54:06.619127 25709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:54:06.633167 25709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:54:06.638662 25709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:54:06.639005 25709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:54:06.659938 25709 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:43839
I20260928 16:54:06.659962 25776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:43839 every 8 connection(s)
I20260928 16:54:06.661939 25709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:54:06.667044 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 25709
I20260928 16:54:06.668035 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:41141
--local_ip_for_outbound_sockets=127.9.93.1
--tserver_master_addrs=127.9.93.62:43839
--webserver_port=44989
--webserver_interface=127.9.93.1
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:34633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:54:06.670823 25777 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:54:06.674903 25777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609: Bootstrap starting.
I20260928 16:54:06.698331 25777 log.cc:824] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609: Log is configured to *not* fsync() on all Append() calls
I20260928 16:54:06.714931 25777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:54:06.715682 25777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609: Bootstrap complete.
I20260928 16:54:06.726531 25777 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9396a025b064ed1b3cf406e8e7ea609" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43839 } }
I20260928 16:54:06.727761 25777 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9396a025b064ed1b3cf406e8e7ea609, State: Initialized, Role: FOLLOWER
I20260928 16:54:06.728554 25777 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9396a025b064ed1b3cf406e8e7ea609" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43839 } }
I20260928 16:54:06.728844 25777 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:54:06.728983 25777 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:54:06.729179 25777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:54:06.734028 25777 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9396a025b064ed1b3cf406e8e7ea609" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43839 } }
I20260928 16:54:06.734741 25777 leader_election.cc:302] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9396a025b064ed1b3cf406e8e7ea609; no voters:
I20260928 16:54:06.735507 25777 leader_election.cc:288] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 16:54:06.735777 25781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 16:54:06.736866 25781 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [term 2 LEADER]: Becoming Leader. State: Replica: a9396a025b064ed1b3cf406e8e7ea609, State: Running, Role: LEADER
I20260928 16:54:06.737555 25781 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9396a025b064ed1b3cf406e8e7ea609" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43839 } }
I20260928 16:54:06.738166 25777 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:54:06.740723 25782 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a9396a025b064ed1b3cf406e8e7ea609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9396a025b064ed1b3cf406e8e7ea609" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43839 } } }
I20260928 16:54:06.741163 25782 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [sys.catalog]: This master's current role is: LEADER
I20260928 16:54:06.741662 25783 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9396a025b064ed1b3cf406e8e7ea609. Latest consensus state: current_term: 2 leader_uuid: "a9396a025b064ed1b3cf406e8e7ea609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9396a025b064ed1b3cf406e8e7ea609" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43839 } } }
I20260928 16:54:06.742002 25783 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [sys.catalog]: This master's current role is: LEADER
I20260928 16:54:06.755764 25787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:54:06.766713 25787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:54:06.768877 25787 catalog_manager.cc:1259] Loaded cluster ID: 4fe20597e2904c32952bb09c2582b129
I20260928 16:54:06.769133 25787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:54:06.774410 25787 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:54:06.776856 25787 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609: Loaded TSK: 0
I20260928 16:54:06.811470 25787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609: Generated new TSK 1
I20260928 16:54:06.812031 25787 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:54:07.102669 25779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:54:07.103032 25779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:54:07.103166 25779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:54:07.111846 25779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:54:07.111989 25779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:54:07.112171 25779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:54:07.123600 25779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:34633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:41141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=44989
--tserver_master_addrs=127.9.93.62:43839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:54:07.125291 25779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:54:07.126897 25779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:54:07.145422 25805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:07.146111 25807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:07.146078 25804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:54:07.146644 25779 server_base.cc:1034] running on GCE node
I20260928 16:54:07.147503 25779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:54:07.148695 25779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:54:07.150019 25779 hybrid_clock.cc:648] HybridClock initialized: now 1790614447149951 us; error 51 us; skew 500 ppm
I20260928 16:54:07.150483 25779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:54:07 dist-test-slave-wzcb krb5kdc[25157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614447, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:54:07.154394 25779 init.cc:377] Logged in from keytab as kudu/127.9.93.1@KRBTEST.COM (short username kudu)
I20260928 16:54:07.156438 25779 webserver.cc:460] Webserver started at http://127.9.93.1:44989/ using document root <none> and password file <none>
I20260928 16:54:07.157363 25779 fs_manager.cc:359] Metadata directory not provided
I20260928 16:54:07.157524 25779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:54:07.164119 25779 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:54:07.167387 25814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:07.169128 25779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:54:07.169318 25779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "af87258b04c348719769cc4ffdda32ed"
format_stamp: "Formatted at 2026-09-28 16:53:43 on dist-test-slave-wzcb"
server_key: "f3895f9b0adc933ba606b5fe1906e9ca"
server_key_iv: "401ff682d7c58b51f3545161983434c8"
server_key_version: "encryptionkey@0"
I20260928 16:54:07.170469 25779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:54:07.201421 25779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:54:07.207926 25779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:54:07.208252 25779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:54:07.209488 25779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:54:07.211216 25779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:54:07.211406 25779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:07.211519 25779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:54:07.211577 25779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:07.244526 25779 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:41141
I20260928 16:54:07.244604 25926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:41141 every 8 connection(s)
I20260928 16:54:07.246189 25779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:54:07.249589 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 25779
I20260928 16:54:07.250475 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:36357
--local_ip_for_outbound_sockets=127.9.93.2
--tserver_master_addrs=127.9.93.62:43839
--webserver_port=40607
--webserver_interface=127.9.93.2
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:34633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:54:07 dist-test-slave-wzcb krb5kdc[25157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614447, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:54:07.277709 25930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:07.249962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.1:51035)
0928 16:54:07.250951 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:07.251008 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:07.253671 (+ 2663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:07.255710 (+ 2039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:07.255724 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:07.256577 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:54:07.257534 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:07.257566 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:07.259535 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:07.259541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:07.261372 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:07.261394 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:07.263068 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:07.263074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:07.263353 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:07.264111 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:54:07.264137 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:54:07.266641 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:54:07.272040 (+ 5399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:54:07.272051 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:54:07.272067 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:54:07.272657 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:54:07.273504 (+ 847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:54:07.273508 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:54:07.273511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:54:07.273795 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:54:07.275115 (+ 1320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:54:07.275123 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:54:07.275619 (+ 496us) client_negotiation.cc:769] Sending connection context
0928 16:54:07.275947 (+ 328us) client_negotiation.cc:240] Negotiation successful
0928 16:54:07.276490 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":372,"threads_started":1}
I20260928 16:54:07.278205 25929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:07.249732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:51035 (local address 127.9.93.62:43839)
0928 16:54:07.250407 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:07.250430 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:07.251178 (+ 748us) server_negotiation.cc:407] Connection header received
0928 16:54:07.254478 (+ 3300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:07.254496 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:07.254965 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:07.255457 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:07.257843 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:07.259265 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:07.261581 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:07.262763 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:07.263004 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:07.267091 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:07.267106 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:07.267134 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:07.267194 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:07.271746 (+ 4552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:07.272920 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:07.272932 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:07.272943 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:07.273259 (+ 316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:07.273976 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:07.273982 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:07.273985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:07.274894 (+ 909us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:07.275136 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:07.276257 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0928 16:54:07.276816 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":286,"threads_started":1}
I20260928 16:54:07.281878 25927 heartbeater.cc:347] Connected to a master server at 127.9.93.62:43839
I20260928 16:54:07.282351 25927 heartbeater.cc:464] Registering TS with master...
I20260928 16:54:07.283504 25927 heartbeater.cc:511] Master 127.9.93.62:43839 requested a full tablet report, sending...
I20260928 16:54:07.286530 25742 ts_manager.cc:194] Registered new tserver with Master: af87258b04c348719769cc4ffdda32ed (127.9.93.1:41141)
I20260928 16:54:07.288362 25742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.1@KRBTEST.COM'} at 127.9.93.1:51035
W20260928 16:54:07.630366 25931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:54:07.630702 25931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:54:07.630837 25931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:54:07.638834 25931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:54:07.638990 25931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:54:07.639159 25931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:54:07.649891 25931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:34633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:36357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=40607
--tserver_master_addrs=127.9.93.62:43839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:54:07.651484 25931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:54:07.653100 25931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:54:07.669708 25936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:07.669740 25937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:07.670802 25939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:54:07.672843 25931 server_base.cc:1034] running on GCE node
I20260928 16:54:07.673628 25931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:54:07.674865 25931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:54:07.676107 25931 hybrid_clock.cc:648] HybridClock initialized: now 1790614447676032 us; error 77 us; skew 500 ppm
I20260928 16:54:07.676488 25931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:54:07 dist-test-slave-wzcb krb5kdc[25157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614447, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:54:07.680163 25931 init.cc:377] Logged in from keytab as kudu/127.9.93.2@KRBTEST.COM (short username kudu)
I20260928 16:54:07.682085 25931 webserver.cc:460] Webserver started at http://127.9.93.2:40607/ using document root <none> and password file <none>
I20260928 16:54:07.682961 25931 fs_manager.cc:359] Metadata directory not provided
I20260928 16:54:07.683110 25931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:54:07.690094 25931 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260928 16:54:07.693807 25946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:07.695833 25931 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:54:07.696056 25931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "0f9d1fc4e00c4e339c566a7e501208d2"
format_stamp: "Formatted at 2026-09-28 16:53:44 on dist-test-slave-wzcb"
server_key: "517dc409e71c298924582b8067522c8e"
server_key_iv: "6cda13483bad7cd8696246a21a854cbd"
server_key_version: "encryptionkey@0"
I20260928 16:54:07.697238 25931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:54:07.724154 25931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:54:07.729787 25931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:54:07.730140 25931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:54:07.731462 25931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:54:07.733379 25931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:54:07.733502 25931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:07.733633 25931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:54:07.733749 25931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:07.769747 25931 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:36357
I20260928 16:54:07.769822 26058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:36357 every 8 connection(s)
I20260928 16:54:07.771523 25931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:54:07.775651 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 25931
I20260928 16:54:07.776525 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:36555
--local_ip_for_outbound_sockets=127.9.93.3
--tserver_master_addrs=127.9.93.62:43839
--webserver_port=41393
--webserver_interface=127.9.93.3
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:34633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:54:07 dist-test-slave-wzcb krb5kdc[25157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614447, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:54:07.794926 25929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:07.774493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:36025 (local address 127.9.93.62:43839)
0928 16:54:07.774709 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:07.774715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:07.776458 (+ 1743us) server_negotiation.cc:407] Connection header received
0928 16:54:07.778584 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:07.778591 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:07.778710 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:07.778808 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:07.780711 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:07.781516 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:07.783296 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:07.783805 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:07.783934 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:07.788196 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:07.788205 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:07.788210 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:07.788262 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:07.790972 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:07.792178 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:07.792183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:07.792186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:07.792335 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:07.792963 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:07.792967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:07.792970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:07.793306 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:07.793454 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:07.794407 (+ 953us) server_negotiation.cc:299] Negotiation successful
0928 16:54:07.794646 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 16:54:07.796649 26061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:07.775157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.2:36025)
0928 16:54:07.776249 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:07.776306 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:07.778235 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:07.779124 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:07.779134 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:07.779794 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:54:07.780472 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:07.780493 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:07.781699 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:07.781703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:07.783101 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:07.783120 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:07.784074 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:07.784082 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:07.784354 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:07.785104 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:54:07.785129 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:54:07.787813 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:54:07.791194 (+ 3381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:54:07.791206 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:54:07.791231 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:54:07.791997 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:54:07.792477 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:54:07.792482 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:54:07.792486 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:54:07.792802 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:54:07.793455 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:54:07.793464 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:54:07.793989 (+ 525us) client_negotiation.cc:769] Sending connection context
0928 16:54:07.794320 (+ 331us) client_negotiation.cc:240] Negotiation successful
0928 16:54:07.795008 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":423,"threads_started":1}
I20260928 16:54:07.798695 26059 heartbeater.cc:347] Connected to a master server at 127.9.93.62:43839
I20260928 16:54:07.799129 26059 heartbeater.cc:464] Registering TS with master...
I20260928 16:54:07.800122 26059 heartbeater.cc:511] Master 127.9.93.62:43839 requested a full tablet report, sending...
I20260928 16:54:07.802189 25742 ts_manager.cc:194] Registered new tserver with Master: 0f9d1fc4e00c4e339c566a7e501208d2 (127.9.93.2:36357)
I20260928 16:54:07.803503 25742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.2@KRBTEST.COM'} at 127.9.93.2:36025
W20260928 16:54:08.147662 26062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:54:08.148012 26062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:54:08.148142 26062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:54:08.156554 26062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:54:08.156709 26062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:54:08.156873 26062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:54:08.168326 26062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:34633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:36555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=41393
--tserver_master_addrs=127.9.93.62:43839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:54:08.169952 26062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:54:08.171687 26062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:54:08.187044 26068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:08.187183 26070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:08.187217 26067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:54:08.188225 26062 server_base.cc:1034] running on GCE node
I20260928 16:54:08.188964 26062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:54:08.190112 26062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:54:08.191455 26062 hybrid_clock.cc:648] HybridClock initialized: now 1790614448191386 us; error 54 us; skew 500 ppm
I20260928 16:54:08.191902 26062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:54:08 dist-test-slave-wzcb krb5kdc[25157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614448, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:54:08.195565 26062 init.cc:377] Logged in from keytab as kudu/127.9.93.3@KRBTEST.COM (short username kudu)
I20260928 16:54:08.197466 26062 webserver.cc:460] Webserver started at http://127.9.93.3:41393/ using document root <none> and password file <none>
I20260928 16:54:08.198280 26062 fs_manager.cc:359] Metadata directory not provided
I20260928 16:54:08.198446 26062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:54:08.205241 26062 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:54:08.208923 26077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:08.210772 26062 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:54:08.210989 26062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "af8fe338838d437ca36bcaf2fc0d89b6"
format_stamp: "Formatted at 2026-09-28 16:53:45 on dist-test-slave-wzcb"
server_key: "ec918ffc3df798e2f3554c756ca54273"
server_key_iv: "bb55c5fd2dc5acf93f75b56a63360d48"
server_key_version: "encryptionkey@0"
I20260928 16:54:08.212263 26062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:54:08.252763 26062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:54:08.258164 26062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:54:08.258487 26062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:54:08.259693 26062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:54:08.261250 26062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:54:08.261351 26062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:08.261462 26062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:54:08.261530 26062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:08.292959 25927 heartbeater.cc:503] Master 127.9.93.62:43839 was elected leader, sending a full tablet report...
I20260928 16:54:08.294168 26062 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:36555
I20260928 16:54:08.294214 26189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:36555 every 8 connection(s)
I20260928 16:54:08.296142 26062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:54:08.299368 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 26062
Sep 28 16:54:08 dist-test-slave-wzcb krb5kdc[25157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614448, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:54:08.319638 26192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:08.300353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35288 (local address 127.9.93.62:43839)
0928 16:54:08.303375 (+ 3022us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:08.303379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:08.303393 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:54:08.303447 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:08.303451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:08.303517 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:08.303618 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:08.305975 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.306916 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.308553 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.309019 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.309148 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:08.312273 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:08.312279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:08.312282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:08.312321 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:08.315875 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:08.316916 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:08.316922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:08.316925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:08.317121 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:08.317740 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:08.317746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:08.317750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:08.318155 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:08.318508 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:08.318851 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 16:54:08.319123 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":2895,"spinlock_wait_cycles":7936,"thread_start_us":2358,"threads_started":1}
I20260928 16:54:08.320647 26194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:08.299180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:50583 (local address 127.9.93.62:43839)
0928 16:54:08.302971 (+ 3791us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:08.302979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:08.303006 (+ 27us) server_negotiation.cc:407] Connection header received
0928 16:54:08.303875 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:08.303879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:08.303929 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:08.303992 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:08.306442 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.307352 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.309046 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.309527 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.309646 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:08.313588 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:08.313595 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:08.313600 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:08.313672 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:08.316892 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:08.317739 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:08.317745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:08.317749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:08.317894 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:08.318350 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:08.318356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:08.318359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:08.318646 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:08.318779 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:08.319976 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0928 16:54:08.320205 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3648,"thread_start_us":3927,"threads_started":1}
I20260928 16:54:08.321695 26193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:08.300055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.3:50583)
0928 16:54:08.301312 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:08.301392 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:08.303541 (+ 2149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:08.304372 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:08.304398 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:08.305251 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:54:08.306193 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:08.306218 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.307524 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.307527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:08.308883 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:08.308897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.309795 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.309804 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:08.310071 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:08.310715 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:54:08.310741 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:54:08.313307 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:54:08.317045 (+ 3738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:54:08.317055 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:54:08.317077 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:54:08.317613 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:54:08.318008 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:54:08.318013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:54:08.318017 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:54:08.318235 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:54:08.318779 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:54:08.318789 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:54:08.319186 (+ 397us) client_negotiation.cc:769] Sending connection context
0928 16:54:08.319523 (+ 337us) client_negotiation.cc:240] Negotiation successful
0928 16:54:08.320113 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":439,"threads_started":1}
I20260928 16:54:08.323360 26190 heartbeater.cc:347] Connected to a master server at 127.9.93.62:43839
I20260928 16:54:08.323755 26190 heartbeater.cc:464] Registering TS with master...
I20260928 16:54:08.324640 26190 heartbeater.cc:511] Master 127.9.93.62:43839 requested a full tablet report, sending...
I20260928 16:54:08.326381 25742 ts_manager.cc:194] Registered new tserver with Master: af8fe338838d437ca36bcaf2fc0d89b6 (127.9.93.3:36555)
I20260928 16:54:08.327266 25742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.3@KRBTEST.COM'} at 127.9.93.3:50583
I20260928 16:54:08.336265 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:54:08.344708 26194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:08.337492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.9.93.62:43839)
0928 16:54:08.337674 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:08.337680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:08.338177 (+ 497us) server_negotiation.cc:407] Connection header received
0928 16:54:08.338473 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:08.338477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:08.338665 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:08.338864 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:54:08.339633 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.340603 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.342318 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.342877 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.343011 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:08.343704 (+ 693us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:54:08.344050 (+ 346us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 16:54:08.344425 (+ 375us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35290: token expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 16:54:08.344838 26194 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35290: token expired
W20260928 16:54:08.345463 25664 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.93.62:43839)
I20260928 16:54:08.360445 26194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:08.346102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35294 (local address 127.9.93.62:43839)
0928 16:54:08.346313 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:08.346318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:08.346338 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:54:08.346499 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:08.346505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:08.346575 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:08.346660 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:08.348381 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.349163 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.350665 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.351116 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.351242 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:08.354154 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:08.354159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:08.354163 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:08.354211 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:08.357315 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:08.358173 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:08.358178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:08.358180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:08.358357 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:08.358795 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:08.358799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:08.358801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:08.359200 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:08.359533 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:08.359851 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 16:54:08.360112 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:54:08.364490 25742 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:54:08.372459 25742 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:54:08.410773 26204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:08.388108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56038 (local address 127.9.93.1:41141)
0928 16:54:08.400787 (+ 12679us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:08.400800 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:08.400846 (+ 46us) server_negotiation.cc:407] Connection header received
0928 16:54:08.400989 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:08.401000 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:08.401376 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:08.401666 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:54:08.403141 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.404757 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.407934 (+ 3177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.409512 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.409634 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:08.409933 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:08.410236 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 16:54:08.410419 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12405,"thread_start_us":14424,"threads_started":1}
I20260928 16:54:08.410831 26201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:08.387152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:41141 (local address 127.0.0.1:56038)
0928 16:54:08.392645 (+ 5493us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:08.392715 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:08.393000 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:08.401737 (+ 8737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:08.401747 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:08.401862 (+ 115us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:54:08.402863 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:08.402892 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.404913 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.404922 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:08.407671 (+ 2749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:08.407690 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.409742 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.409748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:08.409955 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:08.410018 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 16:54:08.410157 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 16:54:08.410392 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5101,"mutex_wait_us":42,"thread_start_us":597,"threads_started":1}
I20260928 16:54:08.412122 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:08.388739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:36555 (local address 127.0.0.1:43204)
0928 16:54:08.403388 (+ 14649us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:08.403403 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:08.403500 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:08.404697 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:08.404702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:08.404728 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:54:08.405160 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:08.405171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.406716 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.406722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:08.408932 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:08.408949 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.411188 (+ 2239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.411194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:08.411411 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:08.411447 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 16:54:08.411540 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 16:54:08.411681 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14519,"mutex_wait_us":323,"thread_start_us":14314,"threads_started":1}
I20260928 16:54:08.412976 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:08.389470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43204 (local address 127.9.93.3:36555)
0928 16:54:08.403731 (+ 14261us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:08.403741 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:08.403777 (+ 36us) server_negotiation.cc:407] Connection header received
0928 16:54:08.403883 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:08.403891 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:08.404168 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:08.404371 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:54:08.405339 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.406531 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.409121 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.410968 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.411116 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:08.411406 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:08.412497 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0928 16:54:08.412675 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13504,"thread_start_us":13346,"threads_started":1}
I20260928 16:54:08.412926 26205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:08.387638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:36357 (local address 127.0.0.1:59558)
0928 16:54:08.403255 (+ 15617us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:08.403276 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:08.403448 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:08.404353 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:08.404358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:08.404387 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:54:08.404814 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:08.404831 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.406757 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.406762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:08.409162 (+ 2400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:08.409178 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.411558 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.411565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:08.411730 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:08.411759 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 16:54:08.411854 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 16:54:08.411995 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15454,"mutex_wait_us":19,"thread_start_us":15918,"threads_started":1}
I20260928 16:54:08.413523 26203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:08.388082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59558 (local address 127.9.93.2:36357)
0928 16:54:08.402601 (+ 14519us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:08.402612 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:08.403425 (+ 813us) server_negotiation.cc:407] Connection header received
0928 16:54:08.403629 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:08.403655 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:08.403980 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:08.404356 (+ 376us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:54:08.405030 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.406579 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.409352 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.411043 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.411160 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:08.411365 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:08.412232 (+ 867us) server_negotiation.cc:299] Negotiation successful
0928 16:54:08.412427 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14306,"thread_start_us":14158,"threads_started":1}
I20260928 16:54:08.416234 25862 tablet_service.cc:1467] Processing CreateTablet for tablet e8ac903b89ce45e48a163df260c096d4 (DEFAULT_TABLE table=test-table [id=87f140f6295e46d29ac12ec4f8dd78e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:54:08.417208 25994 tablet_service.cc:1467] Processing CreateTablet for tablet e8ac903b89ce45e48a163df260c096d4 (DEFAULT_TABLE table=test-table [id=87f140f6295e46d29ac12ec4f8dd78e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:54:08.417208 26125 tablet_service.cc:1467] Processing CreateTablet for tablet e8ac903b89ce45e48a163df260c096d4 (DEFAULT_TABLE table=test-table [id=87f140f6295e46d29ac12ec4f8dd78e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:54:08.418051 25862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8ac903b89ce45e48a163df260c096d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:54:08.418777 26125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8ac903b89ce45e48a163df260c096d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:54:08.418782 25994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8ac903b89ce45e48a163df260c096d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:54:08.432668 26207 tablet_bootstrap.cc:492] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed: Bootstrap starting.
I20260928 16:54:08.437216 26207 tablet_bootstrap.cc:654] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed: Neither blocks nor log segments found. Creating new log.
I20260928 16:54:08.438449 26207 log.cc:824] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed: Log is configured to *not* fsync() on all Append() calls
I20260928 16:54:08.440200 26208 tablet_bootstrap.cc:492] T e8ac903b89ce45e48a163df260c096d4 P 0f9d1fc4e00c4e339c566a7e501208d2: Bootstrap starting.
I20260928 16:54:08.439042 26209 tablet_bootstrap.cc:492] T e8ac903b89ce45e48a163df260c096d4 P af8fe338838d437ca36bcaf2fc0d89b6: Bootstrap starting.
I20260928 16:54:08.444192 26209 tablet_bootstrap.cc:654] T e8ac903b89ce45e48a163df260c096d4 P af8fe338838d437ca36bcaf2fc0d89b6: Neither blocks nor log segments found. Creating new log.
I20260928 16:54:08.444434 26208 tablet_bootstrap.cc:654] T e8ac903b89ce45e48a163df260c096d4 P 0f9d1fc4e00c4e339c566a7e501208d2: Neither blocks nor log segments found. Creating new log.
I20260928 16:54:08.445780 26209 log.cc:824] T e8ac903b89ce45e48a163df260c096d4 P af8fe338838d437ca36bcaf2fc0d89b6: Log is configured to *not* fsync() on all Append() calls
I20260928 16:54:08.446422 26208 log.cc:824] T e8ac903b89ce45e48a163df260c096d4 P 0f9d1fc4e00c4e339c566a7e501208d2: Log is configured to *not* fsync() on all Append() calls
I20260928 16:54:08.446823 26207 tablet_bootstrap.cc:492] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed: No bootstrap required, opened a new log
I20260928 16:54:08.447139 26207 ts_tablet_manager.cc:1397] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20260928 16:54:08.449548 26209 tablet_bootstrap.cc:492] T e8ac903b89ce45e48a163df260c096d4 P af8fe338838d437ca36bcaf2fc0d89b6: No bootstrap required, opened a new log
I20260928 16:54:08.449877 26209 ts_tablet_manager.cc:1397] T e8ac903b89ce45e48a163df260c096d4 P af8fe338838d437ca36bcaf2fc0d89b6: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260928 16:54:08.452041 26208 tablet_bootstrap.cc:492] T e8ac903b89ce45e48a163df260c096d4 P 0f9d1fc4e00c4e339c566a7e501208d2: No bootstrap required, opened a new log
I20260928 16:54:08.452332 26208 ts_tablet_manager.cc:1397] T e8ac903b89ce45e48a163df260c096d4 P 0f9d1fc4e00c4e339c566a7e501208d2: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.001s
I20260928 16:54:08.458747 26207 raft_consensus.cc:348] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af87258b04c348719769cc4ffdda32ed" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 41141 } } peers { permanent_uuid: "af8fe338838d437ca36bcaf2fc0d89b6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36555 } } peers { permanent_uuid: "0f9d1fc4e00c4e339c566a7e501208d2" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 36357 } }
I20260928 16:54:08.459362 26207 raft_consensus.cc:374] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:54:08.459568 26207 raft_consensus.cc:729] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af87258b04c348719769cc4ffdda32ed, State: Initialized, Role: FOLLOWER
I20260928 16:54:08.460419 26207 consensus_queue.cc:260] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af87258b04c348719769cc4ffdda32ed" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 41141 } } peers { permanent_uuid: "af8fe338838d437ca36bcaf2fc0d89b6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36555 } } peers { permanent_uuid: "0f9d1fc4e00c4e339c566a7e501208d2" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 36357 } }
I20260928 16:54:08.460583 26208 raft_consensus.cc:348] T e8ac903b89ce45e48a163df260c096d4 P 0f9d1fc4e00c4e339c566a7e501208d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af87258b04c348719769cc4ffdda32ed" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 41141 } } peers { permanent_uuid: "af8fe338838d437ca36bcaf2fc0d89b6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36555 } } peers { permanent_uuid: "0f9d1fc4e00c4e339c566a7e501208d2" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 36357 } }
I20260928 16:54:08.461086 26208 raft_consensus.cc:374] T e8ac903b89ce45e48a163df260c096d4 P 0f9d1fc4e00c4e339c566a7e501208d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:54:08.461261 26208 raft_consensus.cc:729] T e8ac903b89ce45e48a163df260c096d4 P 0f9d1fc4e00c4e339c566a7e501208d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f9d1fc4e00c4e339c566a7e501208d2, State: Initialized, Role: FOLLOWER
I20260928 16:54:08.461212 26209 raft_consensus.cc:348] T e8ac903b89ce45e48a163df260c096d4 P af8fe338838d437ca36bcaf2fc0d89b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af87258b04c348719769cc4ffdda32ed" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 41141 } } peers { permanent_uuid: "af8fe338838d437ca36bcaf2fc0d89b6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36555 } } peers { permanent_uuid: "0f9d1fc4e00c4e339c566a7e501208d2" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 36357 } }
I20260928 16:54:08.461742 26209 raft_consensus.cc:374] T e8ac903b89ce45e48a163df260c096d4 P af8fe338838d437ca36bcaf2fc0d89b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:54:08.461894 26209 raft_consensus.cc:729] T e8ac903b89ce45e48a163df260c096d4 P af8fe338838d437ca36bcaf2fc0d89b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af8fe338838d437ca36bcaf2fc0d89b6, State: Initialized, Role: FOLLOWER
I20260928 16:54:08.462289 26208 consensus_queue.cc:260] T e8ac903b89ce45e48a163df260c096d4 P 0f9d1fc4e00c4e339c566a7e501208d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af87258b04c348719769cc4ffdda32ed" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 41141 } } peers { permanent_uuid: "af8fe338838d437ca36bcaf2fc0d89b6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36555 } } peers { permanent_uuid: "0f9d1fc4e00c4e339c566a7e501208d2" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 36357 } }
I20260928 16:54:08.462743 26207 ts_tablet_manager.cc:1428] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed: Time spent starting tablet: real 0.015s user 0.006s sys 0.008s
I20260928 16:54:08.462690 26209 consensus_queue.cc:260] T e8ac903b89ce45e48a163df260c096d4 P af8fe338838d437ca36bcaf2fc0d89b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af87258b04c348719769cc4ffdda32ed" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 41141 } } peers { permanent_uuid: "af8fe338838d437ca36bcaf2fc0d89b6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36555 } } peers { permanent_uuid: "0f9d1fc4e00c4e339c566a7e501208d2" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 36357 } }
I20260928 16:54:08.464280 26190 heartbeater.cc:503] Master 127.9.93.62:43839 was elected leader, sending a full tablet report...
I20260928 16:54:08.465131 26059 heartbeater.cc:503] Master 127.9.93.62:43839 was elected leader, sending a full tablet report...
I20260928 16:54:08.465597 26209 ts_tablet_manager.cc:1428] T e8ac903b89ce45e48a163df260c096d4 P af8fe338838d437ca36bcaf2fc0d89b6: Time spent starting tablet: real 0.016s user 0.008s sys 0.005s
I20260928 16:54:08.465631 26208 ts_tablet_manager.cc:1428] T e8ac903b89ce45e48a163df260c096d4 P 0f9d1fc4e00c4e339c566a7e501208d2: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260928 16:54:08.499033 25928 tablet.cc:2369] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:54:08.518116 26213 raft_consensus.cc:482] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:54:08.518496 26213 raft_consensus.cc:504] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af87258b04c348719769cc4ffdda32ed" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 41141 } } peers { permanent_uuid: "af8fe338838d437ca36bcaf2fc0d89b6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36555 } } peers { permanent_uuid: "0f9d1fc4e00c4e339c566a7e501208d2" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 36357 } }
I20260928 16:54:08.520473 26213 leader_election.cc:288] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af8fe338838d437ca36bcaf2fc0d89b6 (127.9.93.3:36555), 0f9d1fc4e00c4e339c566a7e501208d2 (127.9.93.2:36357)
W20260928 16:54:08.523856 26060 tablet.cc:2369] T e8ac903b89ce45e48a163df260c096d4 P 0f9d1fc4e00c4e339c566a7e501208d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:54:08.528064 26203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:08.521098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:56817 (local address 127.9.93.2:36357)
0928 16:54:08.521361 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:08.521367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:08.521951 (+ 584us) server_negotiation.cc:407] Connection header received
0928 16:54:08.522174 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:08.522178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:08.522249 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:08.522330 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:54:08.522952 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.523997 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.526130 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.526956 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.527104 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:08.527208 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:08.527635 (+ 427us) server_negotiation.cc:299] Negotiation successful
0928 16:54:08.527795 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 16:54:08.528060 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:08.520872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:57209 (local address 127.9.93.3:36555)
0928 16:54:08.521111 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:08.521119 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:08.522071 (+ 952us) server_negotiation.cc:407] Connection header received
0928 16:54:08.522239 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:08.522243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:08.522308 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:08.522388 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:54:08.523062 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.523954 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.526089 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.526924 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.527070 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:08.527190 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:08.527626 (+ 436us) server_negotiation.cc:299] Negotiation successful
0928 16:54:08.527779 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 16:54:08.528254 26216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:08.520808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:36555 (local address 127.9.93.1:57209)
0928 16:54:08.521944 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:08.521961 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:08.522105 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:08.522416 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:08.522421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:08.522445 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:54:08.522903 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:08.522912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.524132 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.524138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:08.525894 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:08.525912 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.527191 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.527203 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:08.527447 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:08.527488 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 16:54:08.527581 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 16:54:08.527782 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1017,"mutex_wait_us":15,"thread_start_us":579,"threads_started":1}
I20260928 16:54:08.528306 26217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:08.520790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:36357 (local address 127.9.93.1:56817)
0928 16:54:08.521812 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:08.521831 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:08.522027 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:08.522369 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:08.522373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:08.522399 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:54:08.522797 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:08.522807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.524192 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.524201 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:08.525946 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:08.525961 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.527224 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.527231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:08.527477 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:08.527504 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 16:54:08.527589 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 16:54:08.527791 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":51,"thread_start_us":763,"threads_started":1}
I20260928 16:54:08.529384 26145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8ac903b89ce45e48a163df260c096d4" candidate_uuid: "af87258b04c348719769cc4ffdda32ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8fe338838d437ca36bcaf2fc0d89b6" is_pre_election: true
I20260928 16:54:08.529561 26014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8ac903b89ce45e48a163df260c096d4" candidate_uuid: "af87258b04c348719769cc4ffdda32ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f9d1fc4e00c4e339c566a7e501208d2" is_pre_election: true
I20260928 16:54:08.530087 26014 raft_consensus.cc:2454] T e8ac903b89ce45e48a163df260c096d4 P 0f9d1fc4e00c4e339c566a7e501208d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af87258b04c348719769cc4ffdda32ed in term 0.
I20260928 16:54:08.530088 26145 raft_consensus.cc:2454] T e8ac903b89ce45e48a163df260c096d4 P af8fe338838d437ca36bcaf2fc0d89b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af87258b04c348719769cc4ffdda32ed in term 0.
I20260928 16:54:08.530972 25816 leader_election.cc:302] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9d1fc4e00c4e339c566a7e501208d2, af87258b04c348719769cc4ffdda32ed; no voters:
I20260928 16:54:08.531546 26213 raft_consensus.cc:2781] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:54:08.531723 26213 raft_consensus.cc:482] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:54:08.531837 26213 raft_consensus.cc:3037] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:54:08.534525 26213 raft_consensus.cc:504] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af87258b04c348719769cc4ffdda32ed" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 41141 } } peers { permanent_uuid: "af8fe338838d437ca36bcaf2fc0d89b6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36555 } } peers { permanent_uuid: "0f9d1fc4e00c4e339c566a7e501208d2" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 36357 } }
I20260928 16:54:08.535435 26213 leader_election.cc:288] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed [CANDIDATE]: Term 1 election: Requested vote from peers af8fe338838d437ca36bcaf2fc0d89b6 (127.9.93.3:36555), 0f9d1fc4e00c4e339c566a7e501208d2 (127.9.93.2:36357)
I20260928 16:54:08.536067 26145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8ac903b89ce45e48a163df260c096d4" candidate_uuid: "af87258b04c348719769cc4ffdda32ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8fe338838d437ca36bcaf2fc0d89b6"
I20260928 16:54:08.536209 26014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8ac903b89ce45e48a163df260c096d4" candidate_uuid: "af87258b04c348719769cc4ffdda32ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f9d1fc4e00c4e339c566a7e501208d2"
I20260928 16:54:08.536358 26145 raft_consensus.cc:3037] T e8ac903b89ce45e48a163df260c096d4 P af8fe338838d437ca36bcaf2fc0d89b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:54:08.536476 26014 raft_consensus.cc:3037] T e8ac903b89ce45e48a163df260c096d4 P 0f9d1fc4e00c4e339c566a7e501208d2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:54:08.540105 26014 raft_consensus.cc:2454] T e8ac903b89ce45e48a163df260c096d4 P 0f9d1fc4e00c4e339c566a7e501208d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af87258b04c348719769cc4ffdda32ed in term 1.
I20260928 16:54:08.540105 26145 raft_consensus.cc:2454] T e8ac903b89ce45e48a163df260c096d4 P af8fe338838d437ca36bcaf2fc0d89b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af87258b04c348719769cc4ffdda32ed in term 1.
I20260928 16:54:08.540709 25816 leader_election.cc:302] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0f9d1fc4e00c4e339c566a7e501208d2, af87258b04c348719769cc4ffdda32ed, af8fe338838d437ca36bcaf2fc0d89b6; no voters:
I20260928 16:54:08.541167 26213 raft_consensus.cc:2781] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:54:08.541644 26213 raft_consensus.cc:686] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed [term 1 LEADER]: Becoming Leader. State: Replica: af87258b04c348719769cc4ffdda32ed, State: Running, Role: LEADER
I20260928 16:54:08.542201 26213 consensus_queue.cc:237] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af87258b04c348719769cc4ffdda32ed" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 41141 } } peers { permanent_uuid: "af8fe338838d437ca36bcaf2fc0d89b6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36555 } } peers { permanent_uuid: "0f9d1fc4e00c4e339c566a7e501208d2" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 36357 } }
W20260928 16:54:08.548090 26191 tablet.cc:2369] T e8ac903b89ce45e48a163df260c096d4 P af8fe338838d437ca36bcaf2fc0d89b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:54:08.547570 25739 catalog_manager.cc:5495] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed reported cstate change: term changed from 0 to 1, leader changed from <none> to af87258b04c348719769cc4ffdda32ed (127.9.93.1). New cstate: current_term: 1 leader_uuid: "af87258b04c348719769cc4ffdda32ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af87258b04c348719769cc4ffdda32ed" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 41141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af8fe338838d437ca36bcaf2fc0d89b6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f9d1fc4e00c4e339c566a7e501208d2" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 36357 } health_report { overall_health: UNKNOWN } } }
I20260928 16:54:08.574635 26204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:08.568617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56052 (local address 127.9.93.1:41141)
0928 16:54:08.568852 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:08.568858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:08.568878 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:54:08.569085 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:08.569089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:08.569179 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:08.569346 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:54:08.569738 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.570636 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.572311 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.572739 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.572872 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:08.573417 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:54:08.574022 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:54:08.574218 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:08.574266 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 16:54:08.574371 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 16:54:08.584527 26145 raft_consensus.cc:1261] T e8ac903b89ce45e48a163df260c096d4 P af8fe338838d437ca36bcaf2fc0d89b6 [term 1 FOLLOWER]: Refusing update from remote peer af87258b04c348719769cc4ffdda32ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:54:08.584530 26014 raft_consensus.cc:1261] T e8ac903b89ce45e48a163df260c096d4 P 0f9d1fc4e00c4e339c566a7e501208d2 [term 1 FOLLOWER]: Refusing update from remote peer af87258b04c348719769cc4ffdda32ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:54:08.585671 26213 consensus_queue.cc:1035] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f9d1fc4e00c4e339c566a7e501208d2" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 36357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:54:08.586077 26218 consensus_queue.cc:1035] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "af8fe338838d437ca36bcaf2fc0d89b6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:54:08.601078 26222 mvcc.cc:205] Tried to move back new op lower bound from 7334356781390200832 to 7334356781233197056. Current Snapshot: MvccSnapshot[applied={T|T < 7334356781390200832}]
I20260928 16:54:08.618870 25741 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:54:08.619271 25741 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:54:08.624006 25741 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609: Sending DeleteTablet for 3 replicas of tablet e8ac903b89ce45e48a163df260c096d4
I20260928 16:54:08.625548 25994 tablet_service.cc:1514] Processing DeleteTablet for tablet e8ac903b89ce45e48a163df260c096d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:54:08 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:59558
I20260928 16:54:08.625555 25862 tablet_service.cc:1514] Processing DeleteTablet for tablet e8ac903b89ce45e48a163df260c096d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:54:08 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:56038
I20260928 16:54:08.625851 26125 tablet_service.cc:1514] Processing DeleteTablet for tablet e8ac903b89ce45e48a163df260c096d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:54:08 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:43204
I20260928 16:54:08.626595 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 25779
I20260928 16:54:08.638578 26230 tablet_replica.cc:330] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed: stopping tablet replica
I20260928 16:54:08.639458 26230 raft_consensus.cc:2229] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:54:08.640491 26230 raft_consensus.cc:2258] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:54:08.641230 26231 tablet_replica.cc:330] T e8ac903b89ce45e48a163df260c096d4 P af8fe338838d437ca36bcaf2fc0d89b6: stopping tablet replica
I20260928 16:54:08.641800 26231 raft_consensus.cc:2229] T e8ac903b89ce45e48a163df260c096d4 P af8fe338838d437ca36bcaf2fc0d89b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:54:08.642442 26231 raft_consensus.cc:2258] T e8ac903b89ce45e48a163df260c096d4 P af8fe338838d437ca36bcaf2fc0d89b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:54:08.643626 26230 ts_tablet_manager.cc:1905] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:54:08.645257 26231 ts_tablet_manager.cc:1905] T e8ac903b89ce45e48a163df260c096d4 P af8fe338838d437ca36bcaf2fc0d89b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:54:08.652765 26230 ts_tablet_manager.cc:1918] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:54:08.653028 26231 ts_tablet_manager.cc:1918] T e8ac903b89ce45e48a163df260c096d4 P af8fe338838d437ca36bcaf2fc0d89b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:54:08.653227 26231 log.cc:1196] T e8ac903b89ce45e48a163df260c096d4 P af8fe338838d437ca36bcaf2fc0d89b6: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/e8ac903b89ce45e48a163df260c096d4
I20260928 16:54:08.653402 26230 log.cc:1196] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/wal/wals/e8ac903b89ce45e48a163df260c096d4
I20260928 16:54:08.653872 26231 ts_tablet_manager.cc:1939] T e8ac903b89ce45e48a163df260c096d4 P af8fe338838d437ca36bcaf2fc0d89b6: Deleting consensus metadata
I20260928 16:54:08.654420 26230 ts_tablet_manager.cc:1939] T e8ac903b89ce45e48a163df260c096d4 P af87258b04c348719769cc4ffdda32ed: Deleting consensus metadata
I20260928 16:54:08.656266 25729 catalog_manager.cc:4841] TS af8fe338838d437ca36bcaf2fc0d89b6 (127.9.93.3:36555): tablet e8ac903b89ce45e48a163df260c096d4 (table test-table [id=87f140f6295e46d29ac12ec4f8dd78e7]) successfully deleted
I20260928 16:54:08.656714 25729 catalog_manager.cc:4841] TS af87258b04c348719769cc4ffdda32ed (127.9.93.1:41141): tablet e8ac903b89ce45e48a163df260c096d4 (table test-table [id=87f140f6295e46d29ac12ec4f8dd78e7]) successfully deleted
I20260928 16:54:08.657112 26204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:08.627595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56054 (local address 127.9.93.1:41141)
0928 16:54:08.627787 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:08.627793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:08.628272 (+ 479us) server_negotiation.cc:407] Connection header received
0928 16:54:08.628354 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:08.628361 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:08.628466 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:08.628562 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:08.630829 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.632281 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.634627 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.635520 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.635673 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:08.645365 (+ 9692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:08.645387 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:08.645408 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:08.645467 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:08.650525 (+ 5058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:08.651888 (+ 1363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:08.651900 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:08.651904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:08.652123 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:08.653803 (+ 1680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:08.653814 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:08.653818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:08.654360 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:08.654730 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:08.656485 (+ 1755us) server_negotiation.cc:299] Negotiation successful
0928 16:54:08.656772 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 16:54:08.658026 25922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:54:08.661713 26229 tablet_replica.cc:330] T e8ac903b89ce45e48a163df260c096d4 P 0f9d1fc4e00c4e339c566a7e501208d2: stopping tablet replica
I20260928 16:54:08.662298 26229 raft_consensus.cc:2229] T e8ac903b89ce45e48a163df260c096d4 P 0f9d1fc4e00c4e339c566a7e501208d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:54:08.662822 26229 raft_consensus.cc:2258] T e8ac903b89ce45e48a163df260c096d4 P 0f9d1fc4e00c4e339c566a7e501208d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:54:08.664981 26229 ts_tablet_manager.cc:1905] T e8ac903b89ce45e48a163df260c096d4 P 0f9d1fc4e00c4e339c566a7e501208d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:54:08.670447 26229 ts_tablet_manager.cc:1918] T e8ac903b89ce45e48a163df260c096d4 P 0f9d1fc4e00c4e339c566a7e501208d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:54:08.670682 26229 log.cc:1196] T e8ac903b89ce45e48a163df260c096d4 P 0f9d1fc4e00c4e339c566a7e501208d2: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/e8ac903b89ce45e48a163df260c096d4
I20260928 16:54:08.671161 26229 ts_tablet_manager.cc:1939] T e8ac903b89ce45e48a163df260c096d4 P 0f9d1fc4e00c4e339c566a7e501208d2: Deleting consensus metadata
I20260928 16:54:08.673091 25727 catalog_manager.cc:4841] TS 0f9d1fc4e00c4e339c566a7e501208d2 (127.9.93.2:36357): tablet e8ac903b89ce45e48a163df260c096d4 (table test-table [id=87f140f6295e46d29ac12ec4f8dd78e7]) successfully deleted
I20260928 16:54:08.768848 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 25779
I20260928 16:54:08.791616 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 25931
I20260928 16:54:08.807485 26203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:08.792570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59570 (local address 127.9.93.2:36357)
0928 16:54:08.792750 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:08.792755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:08.792774 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:54:08.792899 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:08.792903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:08.792997 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:08.793082 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:08.794801 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.795793 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.797272 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.797698 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.797800 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:08.801178 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:08.801196 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:08.801234 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:08.801290 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:08.804429 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:08.805269 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:08.805277 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:08.805279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:08.805412 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:08.805838 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:08.805841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:08.805843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:08.806289 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:08.806549 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:08.806915 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 16:54:08.807124 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:54:08.808245 26054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:54:08.918875 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 25931
I20260928 16:54:08.935001 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 26062
I20260928 16:54:08.950382 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:08.935915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43218 (local address 127.9.93.3:36555)
0928 16:54:08.936131 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:08.936136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:08.936159 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:54:08.936280 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:08.936284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:08.936382 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:08.936476 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:08.937970 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.938779 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.940264 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:08.940703 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:08.940820 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:08.943693 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:08.943712 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:08.943730 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:08.943773 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:08.946852 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:08.948014 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:08.948023 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:08.948026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:08.948187 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:08.948629 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:08.948633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:08.948635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:08.949055 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:08.949367 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:08.949760 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 16:54:08.950039 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 16:54:08.951062 26185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:54:09.043275 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 26062
I20260928 16:54:09.057994 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 25709
I20260928 16:54:09.059435 25772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:54:09.180359 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 25709
I20260928 16:54:29.204093 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:43839
--webserver_interface=127.9.93.62
--webserver_port=41865
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:34633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:43839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:54:29.587239 26236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:54:29.587661 26236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:54:29.587788 26236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:54:29.595814 26236 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 16:54:29.595949 26236 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 16:54:29.595990 26236 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 16:54:29.596046 26236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:54:29.596102 26236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:54:29.596177 26236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:54:29.596225 26236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:54:29.596282 26236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:54:29.607199 26236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:34633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:43839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:43839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=41865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:54:29.608796 26236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:54:29.610391 26236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:54:29.624361 26241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:29.624385 26242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:29.624362 26244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:54:29.625396 26236 server_base.cc:1034] running on GCE node
I20260928 16:54:29.626490 26236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:54:29.628398 26236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:54:29.629704 26236 hybrid_clock.cc:648] HybridClock initialized: now 1790614469629589 us; error 74 us; skew 500 ppm
I20260928 16:54:29.630116 26236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:54:29 dist-test-slave-wzcb krb5kdc[25157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614469, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:54:29.633800 26236 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:54:29.635723 26236 webserver.cc:460] Webserver started at http://127.9.93.62:41865/ using document root <none> and password file <none>
I20260928 16:54:29.636649 26236 fs_manager.cc:359] Metadata directory not provided
I20260928 16:54:29.636785 26236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:54:29.643824 26236 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 16:54:29.647747 26251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:29.649566 26236 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:54:29.649793 26236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "a9396a025b064ed1b3cf406e8e7ea609"
format_stamp: "Formatted at 2026-09-28 16:53:43 on dist-test-slave-wzcb"
server_key: "188e7965f23c51a9c935fc6cc1ce8ad8"
server_key_iv: "3b38113c1e2565163687f61457b6f8e1"
server_key_version: "encryptionkey@0"
I20260928 16:54:29.651010 26236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:54:29.681342 26236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:54:29.686714 26236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:54:29.687062 26236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:54:29.708787 26236 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:43839
I20260928 16:54:29.708840 26302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:43839 every 8 connection(s)
I20260928 16:54:29.710721 26236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:54:29.715023 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 26236
I20260928 16:54:29.716252 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:41141
--local_ip_for_outbound_sockets=127.9.93.1
--tserver_master_addrs=127.9.93.62:43839
--webserver_port=44989
--webserver_interface=127.9.93.1
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:34633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:54:29.721087 26303 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:54:29.726569 26303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609: Bootstrap starting.
I20260928 16:54:29.754191 26303 log.cc:824] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609: Log is configured to *not* fsync() on all Append() calls
I20260928 16:54:29.773521 26303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:54:29.774158 26303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609: Bootstrap complete.
I20260928 16:54:29.784302 26303 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9396a025b064ed1b3cf406e8e7ea609" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43839 } }
I20260928 16:54:29.785367 26303 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9396a025b064ed1b3cf406e8e7ea609, State: Initialized, Role: FOLLOWER
I20260928 16:54:29.786181 26303 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9396a025b064ed1b3cf406e8e7ea609" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43839 } }
I20260928 16:54:29.786458 26303 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:54:29.786594 26303 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:54:29.786791 26303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [term 2 FOLLOWER]: Advancing to term 3
I20260928 16:54:29.791358 26303 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9396a025b064ed1b3cf406e8e7ea609" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43839 } }
I20260928 16:54:29.791985 26303 leader_election.cc:302] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9396a025b064ed1b3cf406e8e7ea609; no voters:
I20260928 16:54:29.792695 26303 leader_election.cc:288] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 16:54:29.792853 26307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 16:54:29.795001 26307 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [term 3 LEADER]: Becoming Leader. State: Replica: a9396a025b064ed1b3cf406e8e7ea609, State: Running, Role: LEADER
I20260928 16:54:29.795372 26303 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:54:29.795766 26307 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9396a025b064ed1b3cf406e8e7ea609" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43839 } }
I20260928 16:54:29.798910 26308 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a9396a025b064ed1b3cf406e8e7ea609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9396a025b064ed1b3cf406e8e7ea609" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43839 } } }
I20260928 16:54:29.798949 26309 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9396a025b064ed1b3cf406e8e7ea609. Latest consensus state: current_term: 3 leader_uuid: "a9396a025b064ed1b3cf406e8e7ea609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9396a025b064ed1b3cf406e8e7ea609" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43839 } } }
I20260928 16:54:29.799445 26309 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [sys.catalog]: This master's current role is: LEADER
I20260928 16:54:29.799440 26308 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609 [sys.catalog]: This master's current role is: LEADER
I20260928 16:54:29.810658 26316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:54:29.821027 26316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:54:29.823153 26316 catalog_manager.cc:1259] Loaded cluster ID: 4fe20597e2904c32952bb09c2582b129
I20260928 16:54:29.823406 26316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:54:29.827909 26324 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 16:54:29.828079 26316 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:54:29.830714 26316 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609: Loaded TSK: 0
I20260928 16:54:29.830917 26316 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a9396a025b064ed1b3cf406e8e7ea609: Loaded TSK: 1
F20260928 16:54:29.832276 26316 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdb71f859ad google::LogMessage::Fail() at ??:0
@ 0x7fdb71f8888a google::LogMessage::SendToLog() at ??:0
@ 0x7fdb71f854de google::LogMessage::Flush() at ??:0
@ 0x7fdb71f86a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdb75ce97d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fdb75cd873e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fdb879b41c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fdb879b3783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fdb87a3f2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fdb87a3f0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fdb87a895e9 std::function<>::operator()() at ??:0
@ 0x7fdb879b7cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fdb879b5c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdb87a3d610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fdb87a3d439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fdb74bd1345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdb74bd491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fdb74bd4799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fdb74bafeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdb77e896db start_thread at ??:0
@ 0x7fdb70e1f61f clone at ??:0
W20260928 16:54:30.188620 26305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:54:30.188975 26305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:54:30.189086 26305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:54:30.197153 26305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:54:30.197299 26305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:54:30.197430 26305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:54:30.208263 26305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:34633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:41141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=44989
--tserver_master_addrs=127.9.93.62:43839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:54:30.209818 26305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:54:30.211383 26305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:54:30.226328 26350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:30.226364 26351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:30.226293 26353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:54:30.227399 26305 server_base.cc:1034] running on GCE node
I20260928 16:54:30.228165 26305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:54:30.229300 26305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:54:30.230516 26305 hybrid_clock.cc:648] HybridClock initialized: now 1790614470230474 us; error 103 us; skew 500 ppm
I20260928 16:54:30.230863 26305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:54:30 dist-test-slave-wzcb krb5kdc[25157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614470, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:54:30.234371 26305 init.cc:377] Logged in from keytab as kudu/127.9.93.1@KRBTEST.COM (short username kudu)
I20260928 16:54:30.236145 26305 webserver.cc:460] Webserver started at http://127.9.93.1:44989/ using document root <none> and password file <none>
I20260928 16:54:30.236961 26305 fs_manager.cc:359] Metadata directory not provided
I20260928 16:54:30.237094 26305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:54:30.243659 26305 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 16:54:30.247123 26360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:30.248893 26305 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:54:30.249138 26305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "af87258b04c348719769cc4ffdda32ed"
format_stamp: "Formatted at 2026-09-28 16:53:43 on dist-test-slave-wzcb"
server_key: "f3895f9b0adc933ba606b5fe1906e9ca"
server_key_iv: "401ff682d7c58b51f3545161983434c8"
server_key_version: "encryptionkey@0"
I20260928 16:54:30.250319 26305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:54:30.273998 26305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:54:30.279049 26305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:54:30.279410 26305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:54:30.280617 26305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:54:30.282316 26305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:54:30.282424 26305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:30.282548 26305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:54:30.282621 26305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:30.318002 26305 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:41141
I20260928 16:54:30.318060 26472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:41141 every 8 connection(s)
I20260928 16:54:30.319835 26305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:54:30.323915 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 26305
I20260928 16:54:30.324666 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:36357
--local_ip_for_outbound_sockets=127.9.93.2
--tserver_master_addrs=127.9.93.62:43839
--webserver_port=40607
--webserver_interface=127.9.93.2
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:34633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:54:30.325762 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:30.323272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.1:60207)
0928 16:54:30.324258 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:30.324582 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":366,"threads_started":1}
W20260928 16:54:30.326566 26473 heartbeater.cc:644] Failed to heartbeat to 127.9.93.62:43839 (0 consecutive failures): Network error: Failed to ping master at 127.9.93.62:43839: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
I20260928 16:54:30.327591 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:30.327175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.1:33005)
0928 16:54:30.327376 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:30.327464 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 16:54:30.328526 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:30.328158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.1:55197)
0928 16:54:30.328310 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:30.328391 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 16:54:30.328776 26473 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:54:30.707047 26476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:54:30.707468 26476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:54:30.707633 26476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:54:30.715827 26476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:54:30.715993 26476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:54:30.716161 26476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:54:30.726852 26476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:34633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:36357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=40607
--tserver_master_addrs=127.9.93.62:43839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:54:30.728453 26476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:54:30.730050 26476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:54:30.745340 26481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:30.745265 26482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:30.745303 26484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:54:30.746325 26476 server_base.cc:1034] running on GCE node
I20260928 16:54:30.747066 26476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:54:30.748256 26476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:54:30.749483 26476 hybrid_clock.cc:648] HybridClock initialized: now 1790614470749405 us; error 73 us; skew 500 ppm
I20260928 16:54:30.749838 26476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:54:30 dist-test-slave-wzcb krb5kdc[25157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614470, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:54:30.753424 26476 init.cc:377] Logged in from keytab as kudu/127.9.93.2@KRBTEST.COM (short username kudu)
I20260928 16:54:30.755280 26476 webserver.cc:460] Webserver started at http://127.9.93.2:40607/ using document root <none> and password file <none>
I20260928 16:54:30.756132 26476 fs_manager.cc:359] Metadata directory not provided
I20260928 16:54:30.756269 26476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:54:30.762669 26476 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260928 16:54:30.766189 26491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:30.768036 26476 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:54:30.768240 26476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "0f9d1fc4e00c4e339c566a7e501208d2"
format_stamp: "Formatted at 2026-09-28 16:53:44 on dist-test-slave-wzcb"
server_key: "517dc409e71c298924582b8067522c8e"
server_key_iv: "6cda13483bad7cd8696246a21a854cbd"
server_key_version: "encryptionkey@0"
I20260928 16:54:30.769434 26476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:54:30.785693 26476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:54:30.790639 26476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:54:30.790946 26476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:54:30.792207 26476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:54:30.793803 26476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:54:30.793915 26476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:30.794039 26476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:54:30.794116 26476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:30.827327 26476 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:36357
I20260928 16:54:30.827397 26603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:36357 every 8 connection(s)
I20260928 16:54:30.829077 26476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:54:30.835099 26606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:30.832561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.2:40841)
0928 16:54:30.833464 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:30.833712 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":349,"threads_started":1}
I20260928 16:54:30.835192 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 26476
I20260928 16:54:30.836006 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:36555
--local_ip_for_outbound_sockets=127.9.93.3
--tserver_master_addrs=127.9.93.62:43839
--webserver_port=41393
--webserver_interface=127.9.93.3
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:34633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:54:30.836112 26604 heartbeater.cc:644] Failed to heartbeat to 127.9.93.62:43839 (0 consecutive failures): Network error: Failed to ping master at 127.9.93.62:43839: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
I20260928 16:54:30.837270 26606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:30.836789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.2:60807)
0928 16:54:30.836992 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:30.837098 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 16:54:30.838366 26606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:30.837919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.2:55339)
0928 16:54:30.838092 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:30.838198 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 16:54:30.838732 26604 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:54:31.221927 26607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:54:31.222270 26607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:54:31.222396 26607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:54:31.230293 26607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:54:31.230436 26607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:54:31.230600 26607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:54:31.241467 26607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:34633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:36555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=41393
--tserver_master_addrs=127.9.93.62:43839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:54:31.243013 26607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:54:31.244684 26607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:54:31.260015 26612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:31.260054 26615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:31.260018 26613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:54:31.261168 26607 server_base.cc:1034] running on GCE node
I20260928 16:54:31.261878 26607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:54:31.263005 26607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:54:31.264232 26607 hybrid_clock.cc:648] HybridClock initialized: now 1790614471264151 us; error 78 us; skew 500 ppm
I20260928 16:54:31.264626 26607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:54:31 dist-test-slave-wzcb krb5kdc[25157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614471, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:54:31.273379 26607 init.cc:377] Logged in from keytab as kudu/127.9.93.3@KRBTEST.COM (short username kudu)
I20260928 16:54:31.275396 26607 webserver.cc:460] Webserver started at http://127.9.93.3:41393/ using document root <none> and password file <none>
I20260928 16:54:31.276227 26607 fs_manager.cc:359] Metadata directory not provided
I20260928 16:54:31.276373 26607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:54:31.283164 26607 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 16:54:31.288570 26622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:31.290584 26607 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:54:31.290834 26607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "af8fe338838d437ca36bcaf2fc0d89b6"
format_stamp: "Formatted at 2026-09-28 16:53:45 on dist-test-slave-wzcb"
server_key: "ec918ffc3df798e2f3554c756ca54273"
server_key_iv: "bb55c5fd2dc5acf93f75b56a63360d48"
server_key_version: "encryptionkey@0"
I20260928 16:54:31.292086 26607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:54:31.314548 26607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:54:31.320395 26607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:54:31.320735 26607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:54:31.322132 26607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:54:31.324118 26607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:54:31.324252 26607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:31.324371 26607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:54:31.324448 26607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:31.330363 26637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:31.329434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.1:57813)
0928 16:54:31.329933 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:31.330072 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":227,"threads_started":1}
I20260928 16:54:31.357692 26607 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:36555
I20260928 16:54:31.357743 26735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:36555 every 8 connection(s)
I20260928 16:54:31.359426 26607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:54:31.364993 26738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:31.362716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.3:44721)
0928 16:54:31.363601 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:31.363864 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":328,"threads_started":1}
W20260928 16:54:31.365731 26736 heartbeater.cc:644] Failed to heartbeat to 127.9.93.62:43839 (0 consecutive failures): Network error: Failed to ping master at 127.9.93.62:43839: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
I20260928 16:54:31.366685 26738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:31.366285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.3:38499)
0928 16:54:31.366469 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:31.366554 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 16:54:31.367370 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 26607
I20260928 16:54:31.367847 26738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:31.367428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.3:47769)
0928 16:54:31.367627 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:31.367716 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260928 16:54:31.368160 26736 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 16:54:31.840545 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:31.839581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.2:45973)
0928 16:54:31.840124 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:31.840239 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":248,"threads_started":1}
I20260928 16:54:32.332546 26741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:32.331642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.1:53821)
0928 16:54:32.332245 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:32.332346 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":311,"threads_started":1}
I20260928 16:54:32.370139 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:32.368953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.3:57697)
0928 16:54:32.369549 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:32.369667 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":283,"threads_started":1}
I20260928 16:54:32.842680 26743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:32.841752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.2:56575)
0928 16:54:32.842312 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:32.842441 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":268,"threads_started":1}
I20260928 16:54:33.334527 26744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:33.333632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.1:44573)
0928 16:54:33.334199 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:33.334310 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":303,"threads_started":1}
I20260928 16:54:33.372471 26745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:33.371524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.3:48027)
0928 16:54:33.372148 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:33.372256 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":324,"threads_started":1}
I20260928 16:54:33.844605 26746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:33.843778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.2:39603)
0928 16:54:33.844308 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:33.844407 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":274,"threads_started":1}
I20260928 16:54:34.336550 26747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:34.335689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.1:59471)
0928 16:54:34.336238 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:34.336335 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":279,"threads_started":1}
I20260928 16:54:34.374478 26748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:34.373694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.3:33663)
0928 16:54:34.374195 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:34.374288 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":263,"threads_started":1}
I20260928 16:54:34.846616 26749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:34.845724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.2:57687)
0928 16:54:34.846272 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:34.846372 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":286,"threads_started":1}
I20260928 16:54:35.338706 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:35.337701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.1:44311)
0928 16:54:35.338372 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:35.338481 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":345,"threads_started":1}
I20260928 16:54:35.376499 26751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:35.375590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.3:53897)
0928 16:54:35.376175 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:35.376279 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":300,"threads_started":1}
I20260928 16:54:35.848829 26752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:35.847865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.2:36301)
0928 16:54:35.848513 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:35.848625 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":292,"threads_started":1}
I20260928 16:54:36.340624 26753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:36.339757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.1:39245)
0928 16:54:36.340284 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:36.340436 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":281,"threads_started":1}
I20260928 16:54:36.378600 26754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:36.377742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.3:41755)
0928 16:54:36.378302 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:36.378409 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":289,"threads_started":1}
I20260928 16:54:36.850919 26755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:36.850017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.2:44325)
0928 16:54:36.850603 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:36.850722 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":291,"threads_started":1}
I20260928 16:54:37.342669 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:37.341662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.1:45749)
0928 16:54:37.342285 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:37.342428 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":272,"threads_started":1}
I20260928 16:54:37.380770 26757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:37.379875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.3:38339)
0928 16:54:37.380434 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:37.380576 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":270,"threads_started":1}
I20260928 16:54:37.853087 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:37.852058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.2:60017)
0928 16:54:37.852718 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:37.852820 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":322,"threads_started":1}
I20260928 16:54:38.344683 26759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:38.343840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.1:52761)
0928 16:54:38.344368 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:38.344483 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":269,"threads_started":1}
I20260928 16:54:38.383046 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:38.382056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.3:43071)
0928 16:54:38.382708 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:38.382806 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":316,"threads_started":1}
I20260928 16:54:38.855175 26761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:38.854238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.2:48409)
0928 16:54:38.854869 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:38.854982 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":319,"threads_started":1}
I20260928 16:54:39.346632 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:39.345747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.1:35543)
0928 16:54:39.346327 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:39.346430 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":280,"threads_started":1}
I20260928 16:54:39.385053 26763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:39.384249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.3:33329)
0928 16:54:39.384774 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:39.384872 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":247,"threads_started":1}
I20260928 16:54:39.857175 26764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:39.856258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.2:60945)
0928 16:54:39.856864 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:39.856977 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":278,"threads_started":1}
I20260928 16:54:40.348673 26765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:40.347813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.1:33467)
0928 16:54:40.348371 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:40.348478 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":281,"threads_started":1}
I20260928 16:54:40.387012 26766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:40.386095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.3:60249)
0928 16:54:40.386705 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:40.386801 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":311,"threads_started":1}
I20260928 16:54:40.861886 26767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:40.860934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.2:39889)
0928 16:54:40.861535 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:40.861656 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":312,"threads_started":1}
I20260928 16:54:41.350840 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:41.349955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.1:32861)
0928 16:54:41.350515 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:41.350637 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":234,"threads_started":1}
I20260928 16:54:41.390129 26769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:41.388729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.3:43309)
0928 16:54:41.389674 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:41.389847 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":567,"threads_started":1}
I20260928 16:54:41.864915 26770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:41.863644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.2:54237)
0928 16:54:41.864419 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:41.864581 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":351,"threads_started":1}
I20260928 16:54:42.352912 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:42.352091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.1:46605)
0928 16:54:42.352619 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:42.352729 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":270,"threads_started":1}
I20260928 16:54:42.392622 26772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:42.391692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.3:43103)
0928 16:54:42.392284 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:42.392411 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":300,"threads_started":1}
I20260928 16:54:42.867188 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:42.866339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.2:42381)
0928 16:54:42.866862 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:42.867005 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":253,"threads_started":1}
I20260928 16:54:43.355062 26774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:43.354077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.1:57377)
0928 16:54:43.354734 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:43.354843 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":334,"threads_started":1}
I20260928 16:54:43.394510 26775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:43.393728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.3:33665)
0928 16:54:43.394209 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:43.394315 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":232,"threads_started":1}
I20260928 16:54:43.869393 26776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:43.868459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.2:60413)
0928 16:54:43.869056 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:43.869167 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":283,"threads_started":1}
I20260928 16:54:44.356808 26777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:44.356027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.1:58879)
0928 16:54:44.356500 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:44.356606 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":230,"threads_started":1}
I20260928 16:54:44.396529 26778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:44.395643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.3:45143)
0928 16:54:44.396189 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:44.396317 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":271,"threads_started":1}
I20260928 16:54:44.871381 26779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:44.870491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.2:39727)
0928 16:54:44.871029 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:44.871128 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":278,"threads_started":1}
I20260928 16:54:45.359153 26780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:45.358145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.1:60235)
0928 16:54:45.358732 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:45.358912 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":266,"threads_started":1}
I20260928 16:54:45.398519 26781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:45.397529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.3:46105)
0928 16:54:45.398224 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:45.398308 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":467,"threads_started":1}
I20260928 16:54:45.873626 26782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:45.872706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.2:52721)
0928 16:54:45.873233 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:45.873366 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":255,"threads_started":1}
I20260928 16:54:46.361271 26783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:46.360347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.1:50577)
0928 16:54:46.360922 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:46.361049 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":278,"threads_started":1}
I20260928 16:54:46.375801 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 26305
I20260928 16:54:46.400789 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:46.399813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:43839 (local address 127.9.93.3:38517)
0928 16:54:46.400412 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:46.400544 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.62:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":280,"threads_started":1}
I20260928 16:54:46.405814 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:46.377502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58040 (local address 127.9.93.1:41141)
0928 16:54:46.378462 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:46.378484 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:46.378553 (+ 69us) server_negotiation.cc:407] Connection header received
0928 16:54:46.380428 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:46.380440 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:46.380947 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:46.381298 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:46.383158 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:46.384638 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:46.386143 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:46.387006 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:46.387163 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:46.397560 (+ 10397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:46.397573 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:46.397601 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:46.397665 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:46.401673 (+ 4008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:46.402584 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:46.402593 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:46.402601 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:46.402806 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:46.403354 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:46.403362 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:46.403366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:46.404142 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:46.404424 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:46.404967 (+ 543us) server_negotiation.cc:299] Negotiation successful
0928 16:54:46.405461 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":495,"threads_started":1}
I20260928 16:54:46.407526 26468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:54:46.494518 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 26305
I20260928 16:54:46.512152 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 26476
I20260928 16:54:46.533569 26787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:46.513196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33468 (local address 127.9.93.2:36357)
0928 16:54:46.513818 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:46.513835 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:46.513878 (+ 43us) server_negotiation.cc:407] Connection header received
0928 16:54:46.515639 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:46.515652 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:46.516151 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:46.516538 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:46.518189 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:46.520229 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:46.521621 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:46.522640 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:46.522859 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:46.526390 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:46.526402 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:46.526428 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:46.526494 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:46.529673 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:46.530648 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:46.530655 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:46.530664 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:46.530893 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:46.531362 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:46.531367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:46.531370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:46.532015 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:46.532267 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:46.532830 (+ 563us) server_negotiation.cc:299] Negotiation successful
0928 16:54:46.533268 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":297,"threads_started":1}
I20260928 16:54:46.535255 26599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:54:46.623381 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 26476
I20260928 16:54:46.641085 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 26607
I20260928 16:54:46.661477 26789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:46.642226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37836 (local address 127.9.93.3:36555)
0928 16:54:46.642907 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:46.642923 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:46.642982 (+ 59us) server_negotiation.cc:407] Connection header received
0928 16:54:46.644795 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:46.644808 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:46.645322 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:46.645670 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:46.647248 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:46.648800 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:46.650205 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:46.650986 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:46.651143 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:46.654092 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:46.654101 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:46.654121 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:46.654175 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:46.657321 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:46.658246 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:46.658258 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:46.658270 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:46.658505 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:46.658991 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:46.658995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:46.658997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:46.659765 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:46.660026 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:46.660610 (+ 584us) server_negotiation.cc:299] Negotiation successful
0928 16:54:46.661126 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":324,"threads_started":1}
I20260928 16:54:46.663117 26731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:54:46.751454 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 26607
2026-09-28T16:54:46Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65203 ms)
[----------] 30 tests from SecurityITest (320371 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:54:46 dist-test-slave-wzcb krb5kdc[26793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:54:46 dist-test-slave-wzcb krb5kdc[26793](info): set up 2 sockets
Sep 28 16:54:46 dist-test-slave-wzcb krb5kdc[26793](info): commencing operation
krb5kdc: starting...
W20260928 16:54:47.802098 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:54:47 dist-test-slave-wzcb krb5kdc[26793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:54:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:54:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:54:48.087839 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:38467
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:38467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:54:48.470024 26806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:54:48.470387 26806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:54:48.470486 26806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:54:48.478440 26806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:54:48.478583 26806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:54:48.478632 26806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:54:48.478701 26806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:54:48.478745 26806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:54:48.489655 26806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:38467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:38467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:54:48.491223 26806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:54:48.492821 26806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:54:48.506224 26814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:48.506327 26812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:48.506552 26811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:54:48.507383 26806 server_base.cc:1034] running on GCE node
I20260928 16:54:48.508293 26806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:54:48.510015 26806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:54:48.511267 26806 hybrid_clock.cc:648] HybridClock initialized: now 1790614488511204 us; error 46 us; skew 500 ppm
I20260928 16:54:48.511696 26806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:54:48 dist-test-slave-wzcb krb5kdc[26793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614488, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:54:48.515161 26806 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:54:48.517177 26806 webserver.cc:460] Webserver started at http://127.9.93.62:33145/ using document root <none> and password file <none>
I20260928 16:54:48.518010 26806 fs_manager.cc:359] Metadata directory not provided
I20260928 16:54:48.518136 26806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:54:48.518488 26806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:54:48.522099 26806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "53dff4e3d1cd4d6cb4ff0abe6adccc8f"
format_stamp: "Formatted at 2026-09-28 16:54:48 on dist-test-slave-wzcb"
server_key: "84ba9f5bf7f684b9848b534a323fe0ba"
server_key_iv: "d4917991df2972aa244869790f55856e"
server_key_version: "encryptionkey@0"
I20260928 16:54:48.523001 26806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "53dff4e3d1cd4d6cb4ff0abe6adccc8f"
format_stamp: "Formatted at 2026-09-28 16:54:48 on dist-test-slave-wzcb"
server_key: "84ba9f5bf7f684b9848b534a323fe0ba"
server_key_iv: "d4917991df2972aa244869790f55856e"
server_key_version: "encryptionkey@0"
I20260928 16:54:48.529467 26806 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260928 16:54:48.534034 26821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:48.535784 26806 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:54:48.536010 26806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "53dff4e3d1cd4d6cb4ff0abe6adccc8f"
format_stamp: "Formatted at 2026-09-28 16:54:48 on dist-test-slave-wzcb"
server_key: "84ba9f5bf7f684b9848b534a323fe0ba"
server_key_iv: "d4917991df2972aa244869790f55856e"
server_key_version: "encryptionkey@0"
I20260928 16:54:48.536226 26806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:54:48.574625 26806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:54:48.579542 26806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:54:48.579856 26806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:54:48.600224 26806 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:38467
I20260928 16:54:48.600287 26872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:38467 every 8 connection(s)
I20260928 16:54:48.601948 26806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:54:48.606726 26873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:54:48.609062 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 26806
I20260928 16:54:48.609351 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:54:48.610371 9588 external_mini_cluster.cc:1428] Setting key ae90b571dddcae93aea179601815ca90
I20260928 16:54:48.622584 26873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53dff4e3d1cd4d6cb4ff0abe6adccc8f: Bootstrap starting.
I20260928 16:54:48.627750 26873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53dff4e3d1cd4d6cb4ff0abe6adccc8f: Neither blocks nor log segments found. Creating new log.
Sep 28 16:54:48 dist-test-slave-wzcb krb5kdc[26793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:54:48.629398 26873 log.cc:824] T 00000000000000000000000000000000 P 53dff4e3d1cd4d6cb4ff0abe6adccc8f: Log is configured to *not* fsync() on all Append() calls
I20260928 16:54:48.633543 26873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53dff4e3d1cd4d6cb4ff0abe6adccc8f: No bootstrap required, opened a new log
I20260928 16:54:48.639675 26876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:48.613120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59702 (local address 127.9.93.62:38467)
0928 16:54:48.614136 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:48.614152 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:48.614209 (+ 57us) server_negotiation.cc:407] Connection header received
0928 16:54:48.616266 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:48.616296 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:48.616829 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:48.617241 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:48.619509 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:48.621225 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:48.623487 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:48.624301 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:48.624482 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:48.630705 (+ 6223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:48.630719 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:48.630745 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:48.630809 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:48.634446 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:48.635637 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:48.635646 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:48.635653 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:48.635874 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:48.636358 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:48.636362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:48.636364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:48.637063 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:48.637333 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:48.637869 (+ 536us) server_negotiation.cc:299] Negotiation successful
0928 16:54:48.638318 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":446,"threads_started":1}
I20260928 16:54:48.643774 26873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53dff4e3d1cd4d6cb4ff0abe6adccc8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dff4e3d1cd4d6cb4ff0abe6adccc8f" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 38467 } }
I20260928 16:54:48.644227 26873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53dff4e3d1cd4d6cb4ff0abe6adccc8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:54:48.644330 26873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53dff4e3d1cd4d6cb4ff0abe6adccc8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53dff4e3d1cd4d6cb4ff0abe6adccc8f, State: Initialized, Role: FOLLOWER
I20260928 16:54:48.644968 26873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53dff4e3d1cd4d6cb4ff0abe6adccc8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dff4e3d1cd4d6cb4ff0abe6adccc8f" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 38467 } }
I20260928 16:54:48.645229 26873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53dff4e3d1cd4d6cb4ff0abe6adccc8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:54:48.645375 26873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53dff4e3d1cd4d6cb4ff0abe6adccc8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:54:48.645548 26873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53dff4e3d1cd4d6cb4ff0abe6adccc8f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:54:48.648182 26873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53dff4e3d1cd4d6cb4ff0abe6adccc8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dff4e3d1cd4d6cb4ff0abe6adccc8f" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 38467 } }
I20260928 16:54:48.648782 26873 leader_election.cc:302] T 00000000000000000000000000000000 P 53dff4e3d1cd4d6cb4ff0abe6adccc8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53dff4e3d1cd4d6cb4ff0abe6adccc8f; no voters:
I20260928 16:54:48.649509 26873 leader_election.cc:288] T 00000000000000000000000000000000 P 53dff4e3d1cd4d6cb4ff0abe6adccc8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:54:48.649664 26878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53dff4e3d1cd4d6cb4ff0abe6adccc8f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:54:48.650357 26878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53dff4e3d1cd4d6cb4ff0abe6adccc8f [term 1 LEADER]: Becoming Leader. State: Replica: 53dff4e3d1cd4d6cb4ff0abe6adccc8f, State: Running, Role: LEADER
I20260928 16:54:48.650929 26878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53dff4e3d1cd4d6cb4ff0abe6adccc8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dff4e3d1cd4d6cb4ff0abe6adccc8f" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 38467 } }
I20260928 16:54:48.651810 26873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53dff4e3d1cd4d6cb4ff0abe6adccc8f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:54:48.654081 26880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53dff4e3d1cd4d6cb4ff0abe6adccc8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53dff4e3d1cd4d6cb4ff0abe6adccc8f. Latest consensus state: current_term: 1 leader_uuid: "53dff4e3d1cd4d6cb4ff0abe6adccc8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dff4e3d1cd4d6cb4ff0abe6adccc8f" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 38467 } } }
I20260928 16:54:48.654592 26880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53dff4e3d1cd4d6cb4ff0abe6adccc8f [sys.catalog]: This master's current role is: LEADER
I20260928 16:54:48.654412 26879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53dff4e3d1cd4d6cb4ff0abe6adccc8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53dff4e3d1cd4d6cb4ff0abe6adccc8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dff4e3d1cd4d6cb4ff0abe6adccc8f" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 38467 } } }
I20260928 16:54:48.654692 26879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53dff4e3d1cd4d6cb4ff0abe6adccc8f [sys.catalog]: This master's current role is: LEADER
I20260928 16:54:48.657676 26886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:54:48.663488 26886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:54:48.674289 26886 catalog_manager.cc:1347] Generated new cluster ID: cb3dcc38d91b4b56ab7a84085c1b36b4
I20260928 16:54:48.674422 26886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:54:48.689074 26886 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:54:48.690822 26886 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:54:48.705030 26886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53dff4e3d1cd4d6cb4ff0abe6adccc8f: Generated new TSK 0
I20260928 16:54:48.706444 26886 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
I20260928 16:54:48.779047 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:38467
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:54:49.174252 26899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:54:49.174624 26899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:54:49.174741 26899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:54:49.182791 26899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:54:49.182942 26899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:54:49.183068 26899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:54:49.194561 26899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:38467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:54:49.196237 26899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:54:49.198508 26899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:54:49.219542 26904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:49.219507 26905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:49.219566 26907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:54:49.220633 26899 server_base.cc:1034] running on GCE node
I20260928 16:54:49.221379 26899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:54:49.222520 26899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:54:49.223773 26899 hybrid_clock.cc:648] HybridClock initialized: now 1790614489223699 us; error 47 us; skew 500 ppm
I20260928 16:54:49.224152 26899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:54:49 dist-test-slave-wzcb krb5kdc[26793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614489, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:54:49.228006 26899 init.cc:377] Logged in from keytab as kudu/127.9.93.1@KRBTEST.COM (short username kudu)
I20260928 16:54:49.230257 26899 webserver.cc:460] Webserver started at http://127.9.93.1:39619/ using document root <none> and password file <none>
I20260928 16:54:49.231210 26899 fs_manager.cc:359] Metadata directory not provided
I20260928 16:54:49.231418 26899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:54:49.231801 26899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:54:49.235522 26899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "1dd95705b1ba4be5b61dcbd02368e72e"
format_stamp: "Formatted at 2026-09-28 16:54:49 on dist-test-slave-wzcb"
server_key: "a4e67fe578343cd468e7afadf5583f22"
server_key_iv: "d867811b95865b9a487fd150f5e0b343"
server_key_version: "encryptionkey@0"
I20260928 16:54:49.236501 26899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "1dd95705b1ba4be5b61dcbd02368e72e"
format_stamp: "Formatted at 2026-09-28 16:54:49 on dist-test-slave-wzcb"
server_key: "a4e67fe578343cd468e7afadf5583f22"
server_key_iv: "d867811b95865b9a487fd150f5e0b343"
server_key_version: "encryptionkey@0"
I20260928 16:54:49.243376 26899 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 16:54:49.247738 26914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:49.249605 26899 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260928 16:54:49.249827 26899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "1dd95705b1ba4be5b61dcbd02368e72e"
format_stamp: "Formatted at 2026-09-28 16:54:49 on dist-test-slave-wzcb"
server_key: "a4e67fe578343cd468e7afadf5583f22"
server_key_iv: "d867811b95865b9a487fd150f5e0b343"
server_key_version: "encryptionkey@0"
I20260928 16:54:49.250064 26899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:54:49.272553 26899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:54:49.277680 26899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:54:49.278012 26899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:54:49.279152 26899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:54:49.280778 26899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:54:49.280864 26899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:49.280978 26899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:54:49.281039 26899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:49.313740 26899 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:33247
I20260928 16:54:49.313761 27026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:33247 every 8 connection(s)
I20260928 16:54:49.315294 26899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:54:49.319907 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 26899
I20260928 16:54:49.320176 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:54:49.321084 9588 external_mini_cluster.cc:1428] Setting key 8ecc55cf521e16fe42cd8587df721508
Sep 28 16:54:49 dist-test-slave-wzcb krb5kdc[26793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614489, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:54:49.340898 27029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:49.318795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:58305 (local address 127.9.93.62:38467)
0928 16:54:49.319391 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:49.319396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:49.320922 (+ 1526us) server_negotiation.cc:407] Connection header received
0928 16:54:49.323260 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:49.323266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:49.323402 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:49.323533 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:49.325512 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:49.326308 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:49.328277 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:49.328740 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:49.328888 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:49.333627 (+ 4739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:49.333633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:49.333661 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:49.333709 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:49.336884 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:49.337928 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:49.337933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:49.337936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:49.338095 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:49.338662 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:49.338667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:49.338669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:49.338995 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:49.339152 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:49.340142 (+ 990us) server_negotiation.cc:299] Negotiation successful
0928 16:54:49.340509 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":282,"threads_started":1}
I20260928 16:54:49.342114 27030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:49.319616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:38467 (local address 127.9.93.1:58305)
0928 16:54:49.320703 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:49.320753 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:49.322884 (+ 2131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:49.323870 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:49.323882 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:49.324578 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:54:49.325282 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:49.325306 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:49.326501 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:49.326505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:49.328078 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:49.328098 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:49.329042 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:49.329051 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:49.329388 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:49.330202 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:54:49.330239 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:54:49.333237 (+ 2998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:54:49.337124 (+ 3887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:54:49.337134 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:54:49.337152 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:54:49.337729 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:54:49.338240 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:54:49.338244 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:54:49.338247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:54:49.338514 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:54:49.339157 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:54:49.339168 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:54:49.339659 (+ 491us) client_negotiation.cc:769] Sending connection context
0928 16:54:49.339973 (+ 314us) client_negotiation.cc:240] Negotiation successful
0928 16:54:49.340692 (+ 719us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":471,"threads_started":1}
I20260928 16:54:49.344077 27027 heartbeater.cc:347] Connected to a master server at 127.9.93.62:38467
I20260928 16:54:49.344501 27027 heartbeater.cc:464] Registering TS with master...
I20260928 16:54:49.345614 27027 heartbeater.cc:511] Master 127.9.93.62:38467 requested a full tablet report, sending...
I20260928 16:54:49.348935 26838 ts_manager.cc:194] Registered new tserver with Master: 1dd95705b1ba4be5b61dcbd02368e72e (127.9.93.1:33247)
WARNING: no policy specified for kudu/127.9.93.2@KRBTEST.COM; defaulting to no policy
I20260928 16:54:49.350998 26838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.1@KRBTEST.COM'} at 127.9.93.1:58305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
I20260928 16:54:49.388402 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:38467
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:54:49.766598 27034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:54:49.766963 27034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:54:49.767138 27034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:54:49.775007 27034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:54:49.775178 27034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:54:49.775385 27034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:54:49.786243 27034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:38467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:54:49.787897 27034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:54:49.789443 27034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:54:49.804502 27039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:49.804466 27040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:49.804459 27042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:54:49.805891 27034 server_base.cc:1034] running on GCE node
I20260928 16:54:49.806617 27034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:54:49.807817 27034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:54:49.809084 27034 hybrid_clock.cc:648] HybridClock initialized: now 1790614489808998 us; error 69 us; skew 500 ppm
I20260928 16:54:49.809510 27034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:54:49 dist-test-slave-wzcb krb5kdc[26793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614489, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:54:49.813334 27034 init.cc:377] Logged in from keytab as kudu/127.9.93.2@KRBTEST.COM (short username kudu)
I20260928 16:54:49.815044 27034 webserver.cc:460] Webserver started at http://127.9.93.2:42171/ using document root <none> and password file <none>
I20260928 16:54:49.815913 27034 fs_manager.cc:359] Metadata directory not provided
I20260928 16:54:49.816047 27034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:54:49.816404 27034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:54:49.819851 27034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "5b91ca22bd954483aa3eaa351acdee33"
format_stamp: "Formatted at 2026-09-28 16:54:49 on dist-test-slave-wzcb"
server_key: "6d9595ac66fc19db7707920935525bd1"
server_key_iv: "2404951f71f7b9639d3f83ffdb1ea47e"
server_key_version: "encryptionkey@0"
I20260928 16:54:49.820832 27034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "5b91ca22bd954483aa3eaa351acdee33"
format_stamp: "Formatted at 2026-09-28 16:54:49 on dist-test-slave-wzcb"
server_key: "6d9595ac66fc19db7707920935525bd1"
server_key_iv: "2404951f71f7b9639d3f83ffdb1ea47e"
server_key_version: "encryptionkey@0"
I20260928 16:54:49.827193 27034 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 16:54:49.832157 27049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:49.833938 27034 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:54:49.834177 27034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "5b91ca22bd954483aa3eaa351acdee33"
format_stamp: "Formatted at 2026-09-28 16:54:49 on dist-test-slave-wzcb"
server_key: "6d9595ac66fc19db7707920935525bd1"
server_key_iv: "2404951f71f7b9639d3f83ffdb1ea47e"
server_key_version: "encryptionkey@0"
I20260928 16:54:49.834429 27034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:54:49.855711 27034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:54:49.861196 27034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:54:49.861552 27034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:54:49.862771 27034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:54:49.864564 27034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:54:49.864662 27034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:49.864809 27034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:54:49.864866 27034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:49.899114 27034 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:46853
I20260928 16:54:49.899180 27162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:46853 every 8 connection(s)
I20260928 16:54:49.900696 27034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:54:49.909471 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 27034
I20260928 16:54:49.909659 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:54:49.910161 9588 external_mini_cluster.cc:1428] Setting key 47bfbf864cd633f15d2db8231f7871fb
Sep 28 16:54:49 dist-test-slave-wzcb krb5kdc[26793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614489, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:54:49.926265 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:49.903804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:54155 (local address 127.9.93.62:38467)
0928 16:54:49.904341 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:49.904346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:49.905605 (+ 1259us) server_negotiation.cc:407] Connection header received
0928 16:54:49.907887 (+ 2282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:49.907893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:49.908003 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:49.908112 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:49.910155 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:49.910994 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:49.913404 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:49.914075 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:49.914220 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:49.918505 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:49.918514 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:49.918518 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:49.918573 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:49.922215 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:49.923255 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:49.923261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:49.923265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:49.923478 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:49.924049 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:49.924054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:49.924057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:49.924431 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:49.924558 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:49.925730 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0928 16:54:49.925964 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":258,"threads_started":1}
I20260928 16:54:49.927860 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:49.904509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:38467 (local address 127.9.93.2:54155)
0928 16:54:49.905390 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:49.905446 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:49.907436 (+ 1990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:49.908447 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:49.908458 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:49.909110 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:54:49.909913 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:49.909940 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:49.911192 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:49.911197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:49.913204 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:49.913222 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:49.914358 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:49.914364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:49.914624 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:49.915396 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:54:49.915467 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:54:49.918142 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:54:49.922500 (+ 4358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:54:49.922510 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:54:49.922530 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:54:49.923077 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:54:49.923630 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:54:49.923635 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:54:49.923638 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:54:49.923911 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:54:49.924600 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:54:49.924611 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:54:49.925188 (+ 577us) client_negotiation.cc:769] Sending connection context
0928 16:54:49.925574 (+ 386us) client_negotiation.cc:240] Negotiation successful
0928 16:54:49.926226 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":332,"threads_started":1}
I20260928 16:54:49.929921 27163 heartbeater.cc:347] Connected to a master server at 127.9.93.62:38467
I20260928 16:54:49.930361 27163 heartbeater.cc:464] Registering TS with master...
I20260928 16:54:49.931483 27163 heartbeater.cc:511] Master 127.9.93.62:38467 requested a full tablet report, sending...
I20260928 16:54:49.933634 26838 ts_manager.cc:194] Registered new tserver with Master: 5b91ca22bd954483aa3eaa351acdee33 (127.9.93.2:46853)
I20260928 16:54:49.934839 26838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.2@KRBTEST.COM'} at 127.9.93.2:54155
WARNING: no policy specified for kudu/127.9.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
I20260928 16:54:49.979684 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:38467
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:54:50.355188 27027 heartbeater.cc:503] Master 127.9.93.62:38467 was elected leader, sending a full tablet report...
W20260928 16:54:50.365388 27169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:54:50.365751 27169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:54:50.365899 27169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:54:50.373797 27169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:54:50.373966 27169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:54:50.374128 27169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:54:50.384785 27169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:38467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:54:50.386343 27169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:54:50.387984 27169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:54:50.403486 27174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:50.403585 27177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:50.403545 27175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:54:50.404459 27169 server_base.cc:1034] running on GCE node
I20260928 16:54:50.405246 27169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:54:50.406451 27169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:54:50.407694 27169 hybrid_clock.cc:648] HybridClock initialized: now 1790614490407633 us; error 60 us; skew 500 ppm
I20260928 16:54:50.408087 27169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:54:50 dist-test-slave-wzcb krb5kdc[26793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614490, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:54:50.411808 27169 init.cc:377] Logged in from keytab as kudu/127.9.93.3@KRBTEST.COM (short username kudu)
I20260928 16:54:50.413733 27169 webserver.cc:460] Webserver started at http://127.9.93.3:43001/ using document root <none> and password file <none>
I20260928 16:54:50.414574 27169 fs_manager.cc:359] Metadata directory not provided
I20260928 16:54:50.414712 27169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:54:50.415059 27169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:54:50.418526 27169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "0e278d53543240e9b21a34c80cd3e713"
format_stamp: "Formatted at 2026-09-28 16:54:50 on dist-test-slave-wzcb"
server_key: "cff720663ab044478c0acb9712e7e208"
server_key_iv: "5eb73d328231f38c088796a82f4e36ef"
server_key_version: "encryptionkey@0"
I20260928 16:54:50.419479 27169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "0e278d53543240e9b21a34c80cd3e713"
format_stamp: "Formatted at 2026-09-28 16:54:50 on dist-test-slave-wzcb"
server_key: "cff720663ab044478c0acb9712e7e208"
server_key_iv: "5eb73d328231f38c088796a82f4e36ef"
server_key_version: "encryptionkey@0"
I20260928 16:54:50.425992 27169 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:54:50.430687 27184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:50.432647 27169 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 16:54:50.432873 27169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "0e278d53543240e9b21a34c80cd3e713"
format_stamp: "Formatted at 2026-09-28 16:54:50 on dist-test-slave-wzcb"
server_key: "cff720663ab044478c0acb9712e7e208"
server_key_iv: "5eb73d328231f38c088796a82f4e36ef"
server_key_version: "encryptionkey@0"
I20260928 16:54:50.433125 27169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:54:50.465770 27169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:54:50.471182 27169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:54:50.471575 27169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:54:50.472769 27169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:54:50.474411 27169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:54:50.474501 27169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:50.474647 27169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:54:50.474707 27169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:50.508894 27169 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:36341
I20260928 16:54:50.508975 27296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:36341 every 8 connection(s)
I20260928 16:54:50.510411 27169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:54:50.511193 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 27169
I20260928 16:54:50.511436 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:54:50.512071 9588 external_mini_cluster.cc:1428] Setting key e5dd0a4c109a6e6da620e1bd38cdc822
Sep 28 16:54:50 dist-test-slave-wzcb krb5kdc[26793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614490, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:54:50.534438 27299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:50.513513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:38123 (local address 127.9.93.62:38467)
0928 16:54:50.513995 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:50.514001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:50.515386 (+ 1385us) server_negotiation.cc:407] Connection header received
0928 16:54:50.517616 (+ 2230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:50.517621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:50.517726 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:50.517827 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:50.519747 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.520596 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.522465 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.522903 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.523022 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:50.527174 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:50.527181 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:50.527185 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:50.527234 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:50.530564 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:50.531625 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:50.531629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:50.531632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:50.531785 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:50.532353 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:50.532357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:50.532360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:50.532713 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:50.532868 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:50.533836 (+ 968us) server_negotiation.cc:299] Negotiation successful
0928 16:54:50.534081 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":218,"threads_started":1}
I20260928 16:54:50.535414 27300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:50.514318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:38467 (local address 127.9.93.3:38123)
0928 16:54:50.515179 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:50.515233 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:50.517294 (+ 2061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:50.518097 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:50.518107 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:50.518774 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:54:50.519523 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:50.519546 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.520799 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.520804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:50.522271 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:50.522287 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.523175 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.523181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:50.523494 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:50.524337 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:54:50.524371 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:54:50.526840 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:54:50.530759 (+ 3919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:54:50.530769 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:54:50.530787 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:54:50.531462 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:54:50.531924 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:54:50.531928 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:54:50.531931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:54:50.532195 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:54:50.532882 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:54:50.532889 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:54:50.533345 (+ 456us) client_negotiation.cc:769] Sending connection context
0928 16:54:50.533638 (+ 293us) client_negotiation.cc:240] Negotiation successful
0928 16:54:50.534123 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":341,"threads_started":1}
I20260928 16:54:50.537197 27297 heartbeater.cc:347] Connected to a master server at 127.9.93.62:38467
I20260928 16:54:50.537576 27297 heartbeater.cc:464] Registering TS with master...
I20260928 16:54:50.538578 27297 heartbeater.cc:511] Master 127.9.93.62:38467 requested a full tablet report, sending...
I20260928 16:54:50.540539 26838 ts_manager.cc:194] Registered new tserver with Master: 0e278d53543240e9b21a34c80cd3e713 (127.9.93.3:36341)
I20260928 16:54:50.541746 26838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.3@KRBTEST.COM'} at 127.9.93.3:38123
I20260928 16:54:50.542531 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:54:50.562927 27299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:50.548219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59706 (local address 127.9.93.62:38467)
0928 16:54:50.548419 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:50.548426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:50.549011 (+ 585us) server_negotiation.cc:407] Connection header received
0928 16:54:50.549147 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:50.549151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:50.549248 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:50.549361 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:50.550965 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.551882 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.553575 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.554029 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.554184 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:50.556924 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:50.556932 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:50.556937 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:50.556990 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:50.560111 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:50.561025 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:50.561030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:50.561032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:50.561197 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:50.561550 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:50.561553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:50.561555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:50.561934 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:50.562157 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:50.562446 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 16:54:50.562657 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:54:50.567868 26838 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:54:50.575461 26838 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:54:50.619663 27311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:50.590480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:33247 (local address 127.0.0.1:37968)
0928 16:54:50.607536 (+ 17056us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:50.607586 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:50.607844 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:50.609217 (+ 1373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:50.609226 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:50.609269 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:54:50.609862 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:50.609880 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.612229 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.612236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:50.615332 (+ 3096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:50.615350 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.618417 (+ 3067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.618425 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:50.618693 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:50.618758 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 16:54:50.618928 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 16:54:50.619151 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16711,"mutex_wait_us":64,"thread_start_us":17596,"threads_started":1}
I20260928 16:54:50.620790 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:50.591056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37968 (local address 127.9.93.1:33247)
0928 16:54:50.604716 (+ 13660us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:50.604728 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:50.608143 (+ 3415us) server_negotiation.cc:407] Connection header received
0928 16:54:50.608284 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:50.608293 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:50.608653 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:50.608880 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:54:50.610132 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.612029 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.615519 (+ 3490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.617212 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.617368 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:50.618249 (+ 881us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:50.620227 (+ 1978us) server_negotiation.cc:299] Negotiation successful
0928 16:54:50.620451 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13421,"thread_start_us":13687,"threads_started":1}
I20260928 16:54:50.621742 27312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:50.590718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58122 (local address 127.9.93.3:36341)
0928 16:54:50.606044 (+ 15326us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:50.606056 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:50.608558 (+ 2502us) server_negotiation.cc:407] Connection header received
0928 16:54:50.608893 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:50.608903 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:50.609164 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:50.609353 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:54:50.610482 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.612177 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.617498 (+ 5321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.619190 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.619335 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:50.619597 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:50.621180 (+ 1583us) server_negotiation.cc:299] Negotiation successful
0928 16:54:50.621379 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15093,"thread_start_us":15339,"threads_started":1}
I20260928 16:54:50.622411 27310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:50.590481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:36341 (local address 127.0.0.1:58122)
0928 16:54:50.608417 (+ 17936us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:50.608436 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:50.608672 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:50.609688 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:50.609697 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:50.609725 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:54:50.610223 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:50.610238 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.612382 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.612387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:50.617259 (+ 4872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:50.617276 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.620107 (+ 2831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.620114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:50.620294 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:50.620330 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 16:54:50.620420 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 16:54:50.620569 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17805,"mutex_wait_us":50,"thread_start_us":16494,"threads_started":1}
I20260928 16:54:50.624526 27315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:50.594363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38926 (local address 127.9.93.2:46853)
0928 16:54:50.606810 (+ 12447us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:50.606820 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:50.613753 (+ 6933us) server_negotiation.cc:407] Connection header received
0928 16:54:50.614152 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:50.614162 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:50.614511 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:50.614719 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:54:50.615954 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.617593 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.620251 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.621888 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.621997 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:50.622216 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:50.623968 (+ 1752us) server_negotiation.cc:299] Negotiation successful
0928 16:54:50.624183 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12267,"thread_start_us":12112,"threads_started":1}
I20260928 16:54:50.625283 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:50.594055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:46853 (local address 127.0.0.1:38926)
0928 16:54:50.613601 (+ 19546us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:50.613625 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:50.613899 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:50.615026 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:50.615031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:50.615059 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:54:50.615709 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:50.615724 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.617792 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.617797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:50.620037 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:50.620050 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.623575 (+ 3525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.623580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:50.623721 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:50.623745 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:54:50.624751 (+ 1006us) client_negotiation.cc:240] Negotiation successful
0928 16:54:50.624948 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19378,"thread_start_us":19091,"threads_started":1}
I20260928 16:54:50.626988 27232 tablet_service.cc:1467] Processing CreateTablet for tablet 162082f18410420b81728c8fb6d0ba3b (DEFAULT_TABLE table=test-table [id=8f69dba40cc1432590cb27d401859311]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:54:50.626190 26962 tablet_service.cc:1467] Processing CreateTablet for tablet 162082f18410420b81728c8fb6d0ba3b (DEFAULT_TABLE table=test-table [id=8f69dba40cc1432590cb27d401859311]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:54:50.628701 26962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 162082f18410420b81728c8fb6d0ba3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:54:50.628726 27232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 162082f18410420b81728c8fb6d0ba3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:54:50.629329 27098 tablet_service.cc:1467] Processing CreateTablet for tablet 162082f18410420b81728c8fb6d0ba3b (DEFAULT_TABLE table=test-table [id=8f69dba40cc1432590cb27d401859311]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:54:50.630693 27098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 162082f18410420b81728c8fb6d0ba3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:54:50.644286 27316 tablet_bootstrap.cc:492] T 162082f18410420b81728c8fb6d0ba3b P 5b91ca22bd954483aa3eaa351acdee33: Bootstrap starting.
I20260928 16:54:50.648763 27317 tablet_bootstrap.cc:492] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e: Bootstrap starting.
I20260928 16:54:50.649243 27318 tablet_bootstrap.cc:492] T 162082f18410420b81728c8fb6d0ba3b P 0e278d53543240e9b21a34c80cd3e713: Bootstrap starting.
I20260928 16:54:50.649762 27316 tablet_bootstrap.cc:654] T 162082f18410420b81728c8fb6d0ba3b P 5b91ca22bd954483aa3eaa351acdee33: Neither blocks nor log segments found. Creating new log.
I20260928 16:54:50.651463 27316 log.cc:824] T 162082f18410420b81728c8fb6d0ba3b P 5b91ca22bd954483aa3eaa351acdee33: Log is configured to *not* fsync() on all Append() calls
I20260928 16:54:50.652838 27317 tablet_bootstrap.cc:654] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e: Neither blocks nor log segments found. Creating new log.
I20260928 16:54:50.652863 27318 tablet_bootstrap.cc:654] T 162082f18410420b81728c8fb6d0ba3b P 0e278d53543240e9b21a34c80cd3e713: Neither blocks nor log segments found. Creating new log.
I20260928 16:54:50.654126 27318 log.cc:824] T 162082f18410420b81728c8fb6d0ba3b P 0e278d53543240e9b21a34c80cd3e713: Log is configured to *not* fsync() on all Append() calls
I20260928 16:54:50.654174 27317 log.cc:824] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:54:50.655866 27316 tablet_bootstrap.cc:492] T 162082f18410420b81728c8fb6d0ba3b P 5b91ca22bd954483aa3eaa351acdee33: No bootstrap required, opened a new log
I20260928 16:54:50.656225 27316 ts_tablet_manager.cc:1397] T 162082f18410420b81728c8fb6d0ba3b P 5b91ca22bd954483aa3eaa351acdee33: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260928 16:54:50.657887 27318 tablet_bootstrap.cc:492] T 162082f18410420b81728c8fb6d0ba3b P 0e278d53543240e9b21a34c80cd3e713: No bootstrap required, opened a new log
I20260928 16:54:50.657910 27317 tablet_bootstrap.cc:492] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e: No bootstrap required, opened a new log
I20260928 16:54:50.658195 27318 ts_tablet_manager.cc:1397] T 162082f18410420b81728c8fb6d0ba3b P 0e278d53543240e9b21a34c80cd3e713: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260928 16:54:50.658195 27317 ts_tablet_manager.cc:1397] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 16:54:50.664376 27316 raft_consensus.cc:348] T 162082f18410420b81728c8fb6d0ba3b P 5b91ca22bd954483aa3eaa351acdee33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e278d53543240e9b21a34c80cd3e713" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36341 } } peers { permanent_uuid: "1dd95705b1ba4be5b61dcbd02368e72e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 33247 } } peers { permanent_uuid: "5b91ca22bd954483aa3eaa351acdee33" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 46853 } }
I20260928 16:54:50.664835 27316 raft_consensus.cc:374] T 162082f18410420b81728c8fb6d0ba3b P 5b91ca22bd954483aa3eaa351acdee33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:54:50.664954 27316 raft_consensus.cc:729] T 162082f18410420b81728c8fb6d0ba3b P 5b91ca22bd954483aa3eaa351acdee33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b91ca22bd954483aa3eaa351acdee33, State: Initialized, Role: FOLLOWER
I20260928 16:54:50.665647 27316 consensus_queue.cc:260] T 162082f18410420b81728c8fb6d0ba3b P 5b91ca22bd954483aa3eaa351acdee33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e278d53543240e9b21a34c80cd3e713" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36341 } } peers { permanent_uuid: "1dd95705b1ba4be5b61dcbd02368e72e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 33247 } } peers { permanent_uuid: "5b91ca22bd954483aa3eaa351acdee33" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 46853 } }
I20260928 16:54:50.666945 27163 heartbeater.cc:503] Master 127.9.93.62:38467 was elected leader, sending a full tablet report...
I20260928 16:54:50.667819 27316 ts_tablet_manager.cc:1428] T 162082f18410420b81728c8fb6d0ba3b P 5b91ca22bd954483aa3eaa351acdee33: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260928 16:54:50.669687 27318 raft_consensus.cc:348] T 162082f18410420b81728c8fb6d0ba3b P 0e278d53543240e9b21a34c80cd3e713 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e278d53543240e9b21a34c80cd3e713" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36341 } } peers { permanent_uuid: "1dd95705b1ba4be5b61dcbd02368e72e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 33247 } } peers { permanent_uuid: "5b91ca22bd954483aa3eaa351acdee33" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 46853 } }
I20260928 16:54:50.669687 27317 raft_consensus.cc:348] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e278d53543240e9b21a34c80cd3e713" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36341 } } peers { permanent_uuid: "1dd95705b1ba4be5b61dcbd02368e72e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 33247 } } peers { permanent_uuid: "5b91ca22bd954483aa3eaa351acdee33" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 46853 } }
I20260928 16:54:50.670243 27317 raft_consensus.cc:374] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:54:50.670248 27318 raft_consensus.cc:374] T 162082f18410420b81728c8fb6d0ba3b P 0e278d53543240e9b21a34c80cd3e713 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:54:50.670365 27318 raft_consensus.cc:729] T 162082f18410420b81728c8fb6d0ba3b P 0e278d53543240e9b21a34c80cd3e713 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e278d53543240e9b21a34c80cd3e713, State: Initialized, Role: FOLLOWER
I20260928 16:54:50.670365 27317 raft_consensus.cc:729] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dd95705b1ba4be5b61dcbd02368e72e, State: Initialized, Role: FOLLOWER
I20260928 16:54:50.671119 27318 consensus_queue.cc:260] T 162082f18410420b81728c8fb6d0ba3b P 0e278d53543240e9b21a34c80cd3e713 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e278d53543240e9b21a34c80cd3e713" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36341 } } peers { permanent_uuid: "1dd95705b1ba4be5b61dcbd02368e72e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 33247 } } peers { permanent_uuid: "5b91ca22bd954483aa3eaa351acdee33" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 46853 } }
I20260928 16:54:50.671119 27317 consensus_queue.cc:260] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e278d53543240e9b21a34c80cd3e713" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36341 } } peers { permanent_uuid: "1dd95705b1ba4be5b61dcbd02368e72e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 33247 } } peers { permanent_uuid: "5b91ca22bd954483aa3eaa351acdee33" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 46853 } }
I20260928 16:54:50.673077 27297 heartbeater.cc:503] Master 127.9.93.62:38467 was elected leader, sending a full tablet report...
I20260928 16:54:50.673427 27317 ts_tablet_manager.cc:1428] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260928 16:54:50.673760 27318 ts_tablet_manager.cc:1428] T 162082f18410420b81728c8fb6d0ba3b P 0e278d53543240e9b21a34c80cd3e713: Time spent starting tablet: real 0.015s user 0.017s sys 0.000s
I20260928 16:54:50.700395 27323 raft_consensus.cc:482] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:54:50.700771 27323 raft_consensus.cc:504] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e278d53543240e9b21a34c80cd3e713" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36341 } } peers { permanent_uuid: "1dd95705b1ba4be5b61dcbd02368e72e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 33247 } } peers { permanent_uuid: "5b91ca22bd954483aa3eaa351acdee33" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 46853 } }
I20260928 16:54:50.702837 27323 leader_election.cc:288] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e278d53543240e9b21a34c80cd3e713 (127.9.93.3:36341), 5b91ca22bd954483aa3eaa351acdee33 (127.9.93.2:46853)
I20260928 16:54:50.710443 27312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:50.703263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:48511 (local address 127.9.93.3:36341)
0928 16:54:50.703514 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:50.703522 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:50.704433 (+ 911us) server_negotiation.cc:407] Connection header received
0928 16:54:50.704650 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:50.704655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:50.704723 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:50.704811 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:54:50.705535 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.706430 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.708437 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.709307 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.709452 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:50.709575 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:50.710008 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 16:54:50.710162 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260928 16:54:50.710455 27315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:50.703543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:43093 (local address 127.9.93.2:46853)
0928 16:54:50.703796 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:50.703804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:50.704525 (+ 721us) server_negotiation.cc:407] Connection header received
0928 16:54:50.704697 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:50.704702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:50.704777 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:50.704864 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:54:50.705533 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.706281 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.708451 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.709305 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.709459 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:50.709575 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:50.710017 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 16:54:50.710171 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 16:54:50.710780 27326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:50.703284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:36341 (local address 127.9.93.1:48511)
0928 16:54:50.704298 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:50.704317 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:50.704498 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:50.704847 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:50.704851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:50.704879 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:54:50.705360 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:50.705372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.706578 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.706582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:50.708254 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:50.708270 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.709579 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.709587 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:50.709815 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:50.709856 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 16:54:50.709953 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 16:54:50.710136 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"mutex_wait_us":106,"thread_start_us":718,"threads_started":1}
I20260928 16:54:50.710827 27325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:50.703441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:46853 (local address 127.9.93.1:43093)
0928 16:54:50.704407 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:50.704422 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:50.704558 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:50.704901 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:50.704905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:50.704941 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:54:50.705366 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:50.705375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.706449 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.706453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:50.708236 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:50.708256 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.709636 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.709641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:50.709811 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:50.709855 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 16:54:50.709959 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 16:54:50.710134 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"mutex_wait_us":62,"thread_start_us":455,"threads_started":1}
I20260928 16:54:50.711896 27118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "162082f18410420b81728c8fb6d0ba3b" candidate_uuid: "1dd95705b1ba4be5b61dcbd02368e72e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b91ca22bd954483aa3eaa351acdee33" is_pre_election: true
I20260928 16:54:50.711895 27252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "162082f18410420b81728c8fb6d0ba3b" candidate_uuid: "1dd95705b1ba4be5b61dcbd02368e72e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e278d53543240e9b21a34c80cd3e713" is_pre_election: true
I20260928 16:54:50.712443 27252 raft_consensus.cc:2454] T 162082f18410420b81728c8fb6d0ba3b P 0e278d53543240e9b21a34c80cd3e713 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dd95705b1ba4be5b61dcbd02368e72e in term 0.
I20260928 16:54:50.712435 27118 raft_consensus.cc:2454] T 162082f18410420b81728c8fb6d0ba3b P 5b91ca22bd954483aa3eaa351acdee33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dd95705b1ba4be5b61dcbd02368e72e in term 0.
I20260928 16:54:50.713171 26917 leader_election.cc:302] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dd95705b1ba4be5b61dcbd02368e72e, 5b91ca22bd954483aa3eaa351acdee33; no voters:
I20260928 16:54:50.713575 27323 raft_consensus.cc:2781] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:54:50.713734 27323 raft_consensus.cc:482] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:54:50.713830 27323 raft_consensus.cc:3037] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:54:50.716409 27323 raft_consensus.cc:504] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e278d53543240e9b21a34c80cd3e713" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36341 } } peers { permanent_uuid: "1dd95705b1ba4be5b61dcbd02368e72e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 33247 } } peers { permanent_uuid: "5b91ca22bd954483aa3eaa351acdee33" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 46853 } }
I20260928 16:54:50.717125 27323 leader_election.cc:288] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e [CANDIDATE]: Term 1 election: Requested vote from peers 0e278d53543240e9b21a34c80cd3e713 (127.9.93.3:36341), 5b91ca22bd954483aa3eaa351acdee33 (127.9.93.2:46853)
I20260928 16:54:50.717768 27252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "162082f18410420b81728c8fb6d0ba3b" candidate_uuid: "1dd95705b1ba4be5b61dcbd02368e72e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e278d53543240e9b21a34c80cd3e713"
I20260928 16:54:50.717787 27118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "162082f18410420b81728c8fb6d0ba3b" candidate_uuid: "1dd95705b1ba4be5b61dcbd02368e72e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b91ca22bd954483aa3eaa351acdee33"
I20260928 16:54:50.718045 27252 raft_consensus.cc:3037] T 162082f18410420b81728c8fb6d0ba3b P 0e278d53543240e9b21a34c80cd3e713 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:54:50.718034 27118 raft_consensus.cc:3037] T 162082f18410420b81728c8fb6d0ba3b P 5b91ca22bd954483aa3eaa351acdee33 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:54:50.721609 27252 raft_consensus.cc:2454] T 162082f18410420b81728c8fb6d0ba3b P 0e278d53543240e9b21a34c80cd3e713 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dd95705b1ba4be5b61dcbd02368e72e in term 1.
I20260928 16:54:50.721609 27118 raft_consensus.cc:2454] T 162082f18410420b81728c8fb6d0ba3b P 5b91ca22bd954483aa3eaa351acdee33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dd95705b1ba4be5b61dcbd02368e72e in term 1.
I20260928 16:54:50.722208 26915 leader_election.cc:302] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e278d53543240e9b21a34c80cd3e713, 1dd95705b1ba4be5b61dcbd02368e72e; no voters:
I20260928 16:54:50.722584 27323 raft_consensus.cc:2781] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:54:50.723088 27323 raft_consensus.cc:686] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e [term 1 LEADER]: Becoming Leader. State: Replica: 1dd95705b1ba4be5b61dcbd02368e72e, State: Running, Role: LEADER
I20260928 16:54:50.723680 27323 consensus_queue.cc:237] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e278d53543240e9b21a34c80cd3e713" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36341 } } peers { permanent_uuid: "1dd95705b1ba4be5b61dcbd02368e72e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 33247 } } peers { permanent_uuid: "5b91ca22bd954483aa3eaa351acdee33" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 46853 } }
I20260928 16:54:50.729066 26838 catalog_manager.cc:5495] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dd95705b1ba4be5b61dcbd02368e72e (127.9.93.1). New cstate: current_term: 1 leader_uuid: "1dd95705b1ba4be5b61dcbd02368e72e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e278d53543240e9b21a34c80cd3e713" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dd95705b1ba4be5b61dcbd02368e72e" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 33247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b91ca22bd954483aa3eaa351acdee33" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 46853 } health_report { overall_health: UNKNOWN } } }
I20260928 16:54:50.757714 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:50.751468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37984 (local address 127.9.93.1:33247)
0928 16:54:50.751671 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:50.751676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:50.751696 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:54:50.751927 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:50.751938 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:50.752076 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:50.752223 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:54:50.752668 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.753481 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.755080 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.755568 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.755709 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:50.756284 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:54:50.756961 (+ 677us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:54:50.757107 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:50.757365 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 16:54:50.757468 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 16:54:50.762512 27298 tablet.cc:2369] T 162082f18410420b81728c8fb6d0ba3b P 0e278d53543240e9b21a34c80cd3e713: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:54:50.768172 27118 raft_consensus.cc:1261] T 162082f18410420b81728c8fb6d0ba3b P 5b91ca22bd954483aa3eaa351acdee33 [term 1 FOLLOWER]: Refusing update from remote peer 1dd95705b1ba4be5b61dcbd02368e72e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:54:50.768172 27252 raft_consensus.cc:1261] T 162082f18410420b81728c8fb6d0ba3b P 0e278d53543240e9b21a34c80cd3e713 [term 1 FOLLOWER]: Refusing update from remote peer 1dd95705b1ba4be5b61dcbd02368e72e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:54:50.769618 27323 consensus_queue.cc:1035] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e278d53543240e9b21a34c80cd3e713" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:54:50.770021 27327 consensus_queue.cc:1035] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b91ca22bd954483aa3eaa351acdee33" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 46853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:54:50.803115 26836 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:54:50.803516 26836 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:54:50.808082 26836 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53dff4e3d1cd4d6cb4ff0abe6adccc8f: Sending DeleteTablet for 3 replicas of tablet 162082f18410420b81728c8fb6d0ba3b
I20260928 16:54:50.809580 27232 tablet_service.cc:1514] Processing DeleteTablet for tablet 162082f18410420b81728c8fb6d0ba3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:54:50 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:58122
I20260928 16:54:50.809605 27098 tablet_service.cc:1514] Processing DeleteTablet for tablet 162082f18410420b81728c8fb6d0ba3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:54:50 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:38926
I20260928 16:54:50.809587 26962 tablet_service.cc:1514] Processing DeleteTablet for tablet 162082f18410420b81728c8fb6d0ba3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:54:50 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:37968
I20260928 16:54:50.814328 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 26899
I20260928 16:54:50.822800 27342 tablet_replica.cc:330] T 162082f18410420b81728c8fb6d0ba3b P 5b91ca22bd954483aa3eaa351acdee33: stopping tablet replica
I20260928 16:54:50.824481 27342 raft_consensus.cc:2229] T 162082f18410420b81728c8fb6d0ba3b P 5b91ca22bd954483aa3eaa351acdee33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:54:50.826208 27342 raft_consensus.cc:2258] T 162082f18410420b81728c8fb6d0ba3b P 5b91ca22bd954483aa3eaa351acdee33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:54:50.830152 27342 ts_tablet_manager.cc:1905] T 162082f18410420b81728c8fb6d0ba3b P 5b91ca22bd954483aa3eaa351acdee33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:54:50.836247 27341 tablet_replica.cc:330] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e: stopping tablet replica
I20260928 16:54:50.839931 27341 raft_consensus.cc:2229] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:54:50.840898 27341 raft_consensus.cc:2258] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:54:50.842693 27342 ts_tablet_manager.cc:1918] T 162082f18410420b81728c8fb6d0ba3b P 5b91ca22bd954483aa3eaa351acdee33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:54:50.843066 27342 log.cc:1196] T 162082f18410420b81728c8fb6d0ba3b P 5b91ca22bd954483aa3eaa351acdee33: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/162082f18410420b81728c8fb6d0ba3b
I20260928 16:54:50.843852 27342 ts_tablet_manager.cc:1939] T 162082f18410420b81728c8fb6d0ba3b P 5b91ca22bd954483aa3eaa351acdee33: Deleting consensus metadata
Sep 28 16:54:50 dist-test-slave-wzcb krb5kdc[26793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.1@KRBTEST.COM
I20260928 16:54:50.846679 26824 catalog_manager.cc:4841] TS 5b91ca22bd954483aa3eaa351acdee33 (127.9.93.2:46853): tablet 162082f18410420b81728c8fb6d0ba3b (table test-table [id=8f69dba40cc1432590cb27d401859311]) successfully deleted
I20260928 16:54:50.847836 27340 tablet_replica.cc:330] T 162082f18410420b81728c8fb6d0ba3b P 0e278d53543240e9b21a34c80cd3e713: stopping tablet replica
I20260928 16:54:50.848116 27341 ts_tablet_manager.cc:1905] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:54:50.848556 27340 raft_consensus.cc:2229] T 162082f18410420b81728c8fb6d0ba3b P 0e278d53543240e9b21a34c80cd3e713 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:54:50.849285 27340 raft_consensus.cc:2258] T 162082f18410420b81728c8fb6d0ba3b P 0e278d53543240e9b21a34c80cd3e713 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:54:50.851966 27340 ts_tablet_manager.cc:1905] T 162082f18410420b81728c8fb6d0ba3b P 0e278d53543240e9b21a34c80cd3e713: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:54:50.858177 27341 ts_tablet_manager.cc:1918] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:54:50.858434 27341 log.cc:1196] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-0/wal/wals/162082f18410420b81728c8fb6d0ba3b
I20260928 16:54:50.859005 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:50.816270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37988 (local address 127.9.93.1:33247)
0928 16:54:50.816506 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:50.816514 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:50.828837 (+ 12323us) server_negotiation.cc:407] Connection header received
0928 16:54:50.829484 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:50.829491 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:50.829606 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:50.829692 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:50.831778 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.833177 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.837396 (+ 4219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:50.838255 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:50.838432 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:50.849089 (+ 10657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:50.849107 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:50.849127 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:50.849181 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:50.853841 (+ 4660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:50.854917 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:50.854928 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:50.854932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:50.855215 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:50.855971 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:50.855981 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:50.855984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:50.856540 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:50.856966 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:50.858296 (+ 1330us) server_negotiation.cc:299] Negotiation successful
0928 16:54:50.858617 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 16:54:50.859093 27341 ts_tablet_manager.cc:1939] T 162082f18410420b81728c8fb6d0ba3b P 1dd95705b1ba4be5b61dcbd02368e72e: Deleting consensus metadata
I20260928 16:54:50.860460 27022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:54:50.861411 27340 ts_tablet_manager.cc:1918] T 162082f18410420b81728c8fb6d0ba3b P 0e278d53543240e9b21a34c80cd3e713: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:54:50.861609 27340 log.cc:1196] T 162082f18410420b81728c8fb6d0ba3b P 0e278d53543240e9b21a34c80cd3e713: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/162082f18410420b81728c8fb6d0ba3b
I20260928 16:54:50.862215 27340 ts_tablet_manager.cc:1939] T 162082f18410420b81728c8fb6d0ba3b P 0e278d53543240e9b21a34c80cd3e713: Deleting consensus metadata
I20260928 16:54:50.864387 26822 catalog_manager.cc:4841] TS 0e278d53543240e9b21a34c80cd3e713 (127.9.93.3:36341): tablet 162082f18410420b81728c8fb6d0ba3b (table test-table [id=8f69dba40cc1432590cb27d401859311]) successfully deleted
I20260928 16:54:50.979508 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 26899
I20260928 16:54:50.998184 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 27034
W20260928 16:54:50.998413 26825 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.93.1:33247 (error 108)
I20260928 16:54:51.000577 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:51.000000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:33247 (local address 127.0.0.1:37998)
0928 16:54:51.000200 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:51.000364 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:33247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 16:54:51.000906 26825 catalog_manager.cc:4584] TS 1dd95705b1ba4be5b61dcbd02368e72e (127.9.93.1:33247): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 162082f18410420b81728c8fb6d0ba3b: Network error: Client connection negotiation failed: client connection to 127.9.93.1:33247: connect: Connection refused (error 111)
Sep 28 16:54:51 dist-test-slave-wzcb krb5kdc[26793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.2@KRBTEST.COM
I20260928 16:54:51.017007 27315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:50.999088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38934 (local address 127.9.93.2:46853)
0928 16:54:50.999341 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:50.999349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:50.999400 (+ 51us) server_negotiation.cc:407] Connection header received
0928 16:54:50.999683 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:50.999688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:50.999784 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:50.999881 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:51.001632 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:51.002427 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:51.004060 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:51.004497 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:51.004627 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:51.009726 (+ 5099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:51.009754 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:51.009774 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:51.009838 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:51.013358 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:51.014325 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:51.014335 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:51.014337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:51.014506 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:51.014992 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:51.014997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:51.014999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:51.015566 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:51.015895 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:51.016369 (+ 474us) server_negotiation.cc:299] Negotiation successful
0928 16:54:51.016642 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 16:54:51.017772 27158 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:54:51.054915 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:51.054334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:33247 (local address 127.0.0.1:38010)
0928 16:54:51.054603 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:51.054726 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:33247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260928 16:54:51.056241 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:51.055770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:33247 (local address 127.0.0.1:38024)
0928 16:54:51.055969 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:51.056062 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:33247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 16:54:51.113202 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 27034
I20260928 16:54:51.116798 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:51.116224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:33247 (local address 127.0.0.1:38028)
0928 16:54:51.116498 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:51.116605 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:33247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260928 16:54:51.117854 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:51.117488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:33247 (local address 127.0.0.1:38040)
0928 16:54:51.117639 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:51.117728 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:33247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 16:54:51.128955 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 27169
Sep 28 16:54:51 dist-test-slave-wzcb krb5kdc[26793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.3@KRBTEST.COM
I20260928 16:54:51.145781 27312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:51.129821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58134 (local address 127.9.93.3:36341)
0928 16:54:51.130096 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:51.130102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:51.130124 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:54:51.130364 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:51.130369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:51.130452 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:51.130563 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:51.132052 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:51.132875 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:51.134182 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:51.134628 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:51.134739 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:51.139535 (+ 4796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:51.139556 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:51.139574 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:51.139618 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:51.142570 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:51.143464 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:51.143475 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:51.143479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:51.143630 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:51.144065 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:51.144072 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:51.144074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:51.144492 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:51.144793 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:51.145179 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 16:54:51.145436 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260928 16:54:51.146492 27292 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:54:51.199035 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:51.198359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:33247 (local address 127.0.0.1:38044)
0928 16:54:51.198684 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:51.198809 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:33247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 16:54:51.200609 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:51.199930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:33247 (local address 127.0.0.1:38060)
0928 16:54:51.200248 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:51.200377 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:33247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260928 16:54:51.239728 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 27169
I20260928 16:54:51.254932 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 26806
I20260928 16:54:51.256201 26868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:54:51.372107 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 26806
2026-09-28T16:54:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4628 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T16:54:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:54:51Z Disabled control of system clock
I20260928 16:54:51.459455 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:46751
--webserver_interface=127.9.93.62
--webserver_port=0
--builtin_ntp_servers=127.9.93.20:39267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:46751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:54:51.827242 27352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:54:51.827690 27352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:54:51.827828 27352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:54:51.835812 27352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:54:51.835968 27352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:54:51.836038 27352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:54:51.836117 27352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:54:51.836179 27352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:54:51.846292 27352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:46751
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:46751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:54:51.847921 27352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:54:51.850042 27352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:54:51.863354 27358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:51.863416 27357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:51.863394 27360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:54:51.865087 27352 server_base.cc:1034] running on GCE node
I20260928 16:54:51.866089 27352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:54:51.867924 27352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:54:51.869177 27352 hybrid_clock.cc:648] HybridClock initialized: now 1790614491869111 us; error 57 us; skew 500 ppm
I20260928 16:54:51.869589 27352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:54:51.877947 27352 webserver.cc:460] Webserver started at http://127.9.93.62:33175/ using document root <none> and password file <none>
I20260928 16:54:51.878907 27352 fs_manager.cc:359] Metadata directory not provided
I20260928 16:54:51.879045 27352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:54:51.879448 27352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:54:51.882962 27352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "84f4641058424c1ab3f37aed110dd210"
format_stamp: "Formatted at 2026-09-28 16:54:51 on dist-test-slave-wzcb"
server_key: "cc4dc5dad5d8feebecc2de21f1902cd0"
server_key_iv: "ea99dfb02f9ca964f6c47d4aee6e7dde"
server_key_version: "encryptionkey@0"
I20260928 16:54:51.883996 27352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "84f4641058424c1ab3f37aed110dd210"
format_stamp: "Formatted at 2026-09-28 16:54:51 on dist-test-slave-wzcb"
server_key: "cc4dc5dad5d8feebecc2de21f1902cd0"
server_key_iv: "ea99dfb02f9ca964f6c47d4aee6e7dde"
server_key_version: "encryptionkey@0"
I20260928 16:54:51.890210 27352 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:54:51.894363 27366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:51.896085 27352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:54:51.896308 27352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "84f4641058424c1ab3f37aed110dd210"
format_stamp: "Formatted at 2026-09-28 16:54:51 on dist-test-slave-wzcb"
server_key: "cc4dc5dad5d8feebecc2de21f1902cd0"
server_key_iv: "ea99dfb02f9ca964f6c47d4aee6e7dde"
server_key_version: "encryptionkey@0"
I20260928 16:54:51.896551 27352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:54:51.912796 27352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:54:51.913823 27352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:54:51.914141 27352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:54:51.934334 27352 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:46751
I20260928 16:54:51.934289 27417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:46751 every 8 connection(s)
I20260928 16:54:51.936239 27352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:54:51.937966 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 27352
I20260928 16:54:51.938165 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:54:51.938745 9588 external_mini_cluster.cc:1428] Setting key e667eff0fff2d4c1c6e8f40bdbba06fa
I20260928 16:54:51.941713 27418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:54:51.955096 27418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f4641058424c1ab3f37aed110dd210: Bootstrap starting.
I20260928 16:54:51.955482 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:51.941034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43932 (local address 127.9.93.62:46751)
0928 16:54:51.942262 (+ 1228us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:51.942276 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:51.942338 (+ 62us) server_negotiation.cc:407] Connection header received
0928 16:54:51.944240 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:51.944269 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:51.944899 (+ 630us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:51.945349 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:51.946133 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:51.947971 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:51.949777 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:51.950637 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:51.950874 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:51.952446 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:51.952456 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:51.952476 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:54:51.953188 (+ 712us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:51.953290 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:54:51.953367 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:51.953498 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:51.953769 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 16:54:51.954195 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":428,"threads_started":1}
I20260928 16:54:51.959583 27418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84f4641058424c1ab3f37aed110dd210: Neither blocks nor log segments found. Creating new log.
I20260928 16:54:51.960796 27418 log.cc:824] T 00000000000000000000000000000000 P 84f4641058424c1ab3f37aed110dd210: Log is configured to *not* fsync() on all Append() calls
I20260928 16:54:51.964231 27418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f4641058424c1ab3f37aed110dd210: No bootstrap required, opened a new log
I20260928 16:54:51.972616 27418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84f4641058424c1ab3f37aed110dd210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f4641058424c1ab3f37aed110dd210" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46751 } }
I20260928 16:54:51.973071 27418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84f4641058424c1ab3f37aed110dd210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:54:51.973184 27418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84f4641058424c1ab3f37aed110dd210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84f4641058424c1ab3f37aed110dd210, State: Initialized, Role: FOLLOWER
I20260928 16:54:51.973850 27418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84f4641058424c1ab3f37aed110dd210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f4641058424c1ab3f37aed110dd210" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46751 } }
I20260928 16:54:51.974100 27418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84f4641058424c1ab3f37aed110dd210 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:54:51.974236 27418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84f4641058424c1ab3f37aed110dd210 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:54:51.974406 27418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84f4641058424c1ab3f37aed110dd210 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:54:51.976754 27418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84f4641058424c1ab3f37aed110dd210 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f4641058424c1ab3f37aed110dd210" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46751 } }
I20260928 16:54:51.977342 27418 leader_election.cc:302] T 00000000000000000000000000000000 P 84f4641058424c1ab3f37aed110dd210 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84f4641058424c1ab3f37aed110dd210; no voters:
I20260928 16:54:51.978102 27418 leader_election.cc:288] T 00000000000000000000000000000000 P 84f4641058424c1ab3f37aed110dd210 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:54:51.978255 27423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84f4641058424c1ab3f37aed110dd210 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:54:51.978958 27423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84f4641058424c1ab3f37aed110dd210 [term 1 LEADER]: Becoming Leader. State: Replica: 84f4641058424c1ab3f37aed110dd210, State: Running, Role: LEADER
I20260928 16:54:51.979612 27423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84f4641058424c1ab3f37aed110dd210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f4641058424c1ab3f37aed110dd210" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46751 } }
I20260928 16:54:51.980773 27418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84f4641058424c1ab3f37aed110dd210 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:54:51.982962 27424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f4641058424c1ab3f37aed110dd210 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84f4641058424c1ab3f37aed110dd210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f4641058424c1ab3f37aed110dd210" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46751 } } }
I20260928 16:54:51.983460 27424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f4641058424c1ab3f37aed110dd210 [sys.catalog]: This master's current role is: LEADER
I20260928 16:54:51.983274 27425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f4641058424c1ab3f37aed110dd210 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84f4641058424c1ab3f37aed110dd210. Latest consensus state: current_term: 1 leader_uuid: "84f4641058424c1ab3f37aed110dd210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f4641058424c1ab3f37aed110dd210" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46751 } } }
I20260928 16:54:51.983669 27425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f4641058424c1ab3f37aed110dd210 [sys.catalog]: This master's current role is: LEADER
I20260928 16:54:51.986546 27431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:54:51.992936 27431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:54:52.004381 27431 catalog_manager.cc:1347] Generated new cluster ID: c1bc5fe1bf44433f908531471494c89e
I20260928 16:54:52.004529 27431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:54:52.017709 27431 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:54:52.019860 27431 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:54:52.028133 27431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84f4641058424c1ab3f37aed110dd210: Generated new TSK 0
I20260928 16:54:52.029398 27431 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:54:52.045305 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:46751
--builtin_ntp_servers=127.9.93.20:39267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:54:52.456595 27442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:54:52.456924 27442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:54:52.457058 27442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:54:52.465018 27442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:54:52.465147 27442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:54:52.465289 27442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:54:52.475553 27442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:46751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:54:52.477075 27442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:54:52.479154 27442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:54:52.494170 27447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:52.494122 27450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:52.494129 27448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:54:52.495105 27442 server_base.cc:1034] running on GCE node
I20260928 16:54:52.495849 27442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:54:52.497001 27442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:54:52.498224 27442 hybrid_clock.cc:648] HybridClock initialized: now 1790614492498156 us; error 52 us; skew 500 ppm
I20260928 16:54:52.498588 27442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:54:52.501282 27442 webserver.cc:460] Webserver started at http://127.9.93.1:42343/ using document root <none> and password file <none>
I20260928 16:54:52.502135 27442 fs_manager.cc:359] Metadata directory not provided
I20260928 16:54:52.502256 27442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:54:52.502609 27442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:54:52.506120 27442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "0f3c9f00f80644ea936051352ba38ca4"
format_stamp: "Formatted at 2026-09-28 16:54:52 on dist-test-slave-wzcb"
server_key: "44f0551e4e89f210fb86646297cf36b9"
server_key_iv: "9f3231099ba8fe0ee8cedb1d25582830"
server_key_version: "encryptionkey@0"
I20260928 16:54:52.507108 27442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "0f3c9f00f80644ea936051352ba38ca4"
format_stamp: "Formatted at 2026-09-28 16:54:52 on dist-test-slave-wzcb"
server_key: "44f0551e4e89f210fb86646297cf36b9"
server_key_iv: "9f3231099ba8fe0ee8cedb1d25582830"
server_key_version: "encryptionkey@0"
I20260928 16:54:52.513053 27442 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:54:52.517073 27456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:52.518684 27442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:54:52.518874 27442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "0f3c9f00f80644ea936051352ba38ca4"
format_stamp: "Formatted at 2026-09-28 16:54:52 on dist-test-slave-wzcb"
server_key: "44f0551e4e89f210fb86646297cf36b9"
server_key_iv: "9f3231099ba8fe0ee8cedb1d25582830"
server_key_version: "encryptionkey@0"
I20260928 16:54:52.519104 27442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:54:52.543650 27442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:54:52.544624 27442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:54:52.544965 27442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:54:52.546137 27442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:54:52.547956 27442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:54:52.548051 27442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:52.548164 27442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:54:52.548242 27442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:52.581326 27442 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:37553
I20260928 16:54:52.581398 27568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:37553 every 8 connection(s)
I20260928 16:54:52.582796 27442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:54:52.583209 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 27442
I20260928 16:54:52.583424 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:54:52.584072 9588 external_mini_cluster.cc:1428] Setting key 6eda7f3464a3d83ad1ac4e48bde51c93
I20260928 16:54:52.587689 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:46751
--builtin_ntp_servers=127.9.93.20:39267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:54:52.598840 27571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:52.585717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:46755 (local address 127.9.93.62:46751)
0928 16:54:52.586260 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:52.586267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:52.590007 (+ 3740us) server_negotiation.cc:407] Connection header received
0928 16:54:52.590080 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:52.590083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:52.590158 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:52.590247 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:52.592038 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:52.592768 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:52.595296 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:52.595791 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:52.595903 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:52.597791 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:52.597797 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:52.597801 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:54:52.597827 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:52.597876 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:54:52.597891 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:52.597978 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:52.598324 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 16:54:52.598463 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":396,"threads_started":1}
I20260928 16:54:52.600031 27572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:52.586509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:46751 (local address 127.9.93.1:46755)
0928 16:54:52.587747 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:52.587810 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:52.589638 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:52.590629 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:52.590643 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:52.590801 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:54:52.591828 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:52.591860 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:52.592959 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:52.592964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:52.595083 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:52.595104 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:52.596072 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:52.596080 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:52.596421 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:52.597278 (+ 857us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:54:52.597281 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:54:52.597350 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:54:52.597486 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:54:52.598000 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:54:52.598012 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:54:52.598021 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 16:54:52.598250 (+ 229us) client_negotiation.cc:240] Negotiation successful
0928 16:54:52.598663 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":409,"threads_started":1}
I20260928 16:54:52.602221 27569 heartbeater.cc:347] Connected to a master server at 127.9.93.62:46751
I20260928 16:54:52.602670 27569 heartbeater.cc:464] Registering TS with master...
I20260928 16:54:52.603868 27569 heartbeater.cc:511] Master 127.9.93.62:46751 requested a full tablet report, sending...
I20260928 16:54:52.606855 27383 ts_manager.cc:194] Registered new tserver with Master: 0f3c9f00f80644ea936051352ba38ca4 (127.9.93.1:37553)
I20260928 16:54:52.608994 27383 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.93.1:46755
W20260928 16:54:52.968353 27573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:54:52.968706 27573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:54:52.968854 27573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:54:52.976609 27573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:54:52.976765 27573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:54:52.976919 27573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:54:52.987039 27573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:46751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:54:52.988570 27573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:54:52.990686 27573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:54:53.005879 27578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:53.005849 27579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:53.005914 27581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:54:53.006630 27573 server_base.cc:1034] running on GCE node
I20260928 16:54:53.007494 27573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:54:53.008684 27573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:54:53.009932 27573 hybrid_clock.cc:648] HybridClock initialized: now 1790614493009863 us; error 55 us; skew 500 ppm
I20260928 16:54:53.010315 27573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:54:53.013087 27573 webserver.cc:460] Webserver started at http://127.9.93.2:36893/ using document root <none> and password file <none>
I20260928 16:54:53.013919 27573 fs_manager.cc:359] Metadata directory not provided
I20260928 16:54:53.014036 27573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:54:53.014371 27573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:54:53.017966 27573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "b904ee3a24cc4cc8909b2b381d642d35"
format_stamp: "Formatted at 2026-09-28 16:54:53 on dist-test-slave-wzcb"
server_key: "3d99e70ddf2ab690c71d7ebb31d80c8b"
server_key_iv: "fcda7c878111d3ee10043e292ffdc0ba"
server_key_version: "encryptionkey@0"
I20260928 16:54:53.018929 27573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "b904ee3a24cc4cc8909b2b381d642d35"
format_stamp: "Formatted at 2026-09-28 16:54:53 on dist-test-slave-wzcb"
server_key: "3d99e70ddf2ab690c71d7ebb31d80c8b"
server_key_iv: "fcda7c878111d3ee10043e292ffdc0ba"
server_key_version: "encryptionkey@0"
I20260928 16:54:53.025373 27573 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:54:53.029696 27587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:53.031608 27573 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:54:53.031872 27573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "b904ee3a24cc4cc8909b2b381d642d35"
format_stamp: "Formatted at 2026-09-28 16:54:53 on dist-test-slave-wzcb"
server_key: "3d99e70ddf2ab690c71d7ebb31d80c8b"
server_key_iv: "fcda7c878111d3ee10043e292ffdc0ba"
server_key_version: "encryptionkey@0"
I20260928 16:54:53.032138 27573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:54:53.059163 27573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:54:53.060495 27573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:54:53.060837 27573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:54:53.062186 27573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:54:53.064062 27573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:54:53.064178 27573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:53.064313 27573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:54:53.064399 27573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:53.098315 27573 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:42623
I20260928 16:54:53.098394 27699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:42623 every 8 connection(s)
I20260928 16:54:53.100095 27573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:54:53.102635 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 27573
I20260928 16:54:53.102849 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:54:53.103466 9588 external_mini_cluster.cc:1428] Setting key 17b3cd27f5009cbaed3754911bf226a1
I20260928 16:54:53.106931 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:46751
--builtin_ntp_servers=127.9.93.20:39267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:54:53.116456 27702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:53.103403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:41223 (local address 127.9.93.62:46751)
0928 16:54:53.103943 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:53.103950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:53.105770 (+ 1820us) server_negotiation.cc:407] Connection header received
0928 16:54:53.108061 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:53.108068 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:53.108163 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:53.108292 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:53.109653 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.110656 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.112785 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.113474 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.113628 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:53.115417 (+ 1789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:53.115425 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:53.115430 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:54:53.115462 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:53.115521 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:54:53.115539 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:53.115638 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:53.115964 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 16:54:53.116094 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":265,"threads_started":1}
I20260928 16:54:53.117949 27703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:53.104142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:46751 (local address 127.9.93.2:41223)
0928 16:54:53.105590 (+ 1448us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:53.105657 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:53.107712 (+ 2055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:53.108583 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:53.108594 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:53.108717 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:54:53.109437 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:53.109458 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.110902 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.110907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:53.112545 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:53.112567 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.113831 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.113840 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:53.114116 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:53.114946 (+ 830us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:54:53.114949 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:54:53.115005 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:54:53.115120 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:54:53.115685 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:54:53.115698 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:54:53.115706 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:54:53.115930 (+ 224us) client_negotiation.cc:240] Negotiation successful
0928 16:54:53.116377 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"thread_start_us":531,"threads_started":1}
I20260928 16:54:53.119798 27700 heartbeater.cc:347] Connected to a master server at 127.9.93.62:46751
I20260928 16:54:53.120208 27700 heartbeater.cc:464] Registering TS with master...
I20260928 16:54:53.121376 27700 heartbeater.cc:511] Master 127.9.93.62:46751 requested a full tablet report, sending...
I20260928 16:54:53.123507 27383 ts_manager.cc:194] Registered new tserver with Master: b904ee3a24cc4cc8909b2b381d642d35 (127.9.93.2:42623)
I20260928 16:54:53.124513 27383 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.93.2:41223
W20260928 16:54:53.481597 27704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:54:53.481943 27704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:54:53.482075 27704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:54:53.490021 27704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:54:53.490150 27704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:54:53.490300 27704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:54:53.500161 27704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:46751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:54:53.501731 27704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:54:53.503960 27704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:54:53.519555 27709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:53.519519 27712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:53.519578 27710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:54:53.520483 27704 server_base.cc:1034] running on GCE node
I20260928 16:54:53.521284 27704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:54:53.522441 27704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:54:53.523693 27704 hybrid_clock.cc:648] HybridClock initialized: now 1790614493523599 us; error 86 us; skew 500 ppm
I20260928 16:54:53.524171 27704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:54:53.526901 27704 webserver.cc:460] Webserver started at http://127.9.93.3:42289/ using document root <none> and password file <none>
I20260928 16:54:53.527822 27704 fs_manager.cc:359] Metadata directory not provided
I20260928 16:54:53.527962 27704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:54:53.528318 27704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:54:53.531780 27704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "a0047951409c483b95417ad48c87d8a6"
format_stamp: "Formatted at 2026-09-28 16:54:53 on dist-test-slave-wzcb"
server_key: "ee59b4bd07af27f88d500fcf94325128"
server_key_iv: "884c0c7871424557505136cb22518226"
server_key_version: "encryptionkey@0"
I20260928 16:54:53.532740 27704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "a0047951409c483b95417ad48c87d8a6"
format_stamp: "Formatted at 2026-09-28 16:54:53 on dist-test-slave-wzcb"
server_key: "ee59b4bd07af27f88d500fcf94325128"
server_key_iv: "884c0c7871424557505136cb22518226"
server_key_version: "encryptionkey@0"
I20260928 16:54:53.539215 27704 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 16:54:53.543880 27718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:53.545761 27704 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:54:53.546021 27704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "a0047951409c483b95417ad48c87d8a6"
format_stamp: "Formatted at 2026-09-28 16:54:53 on dist-test-slave-wzcb"
server_key: "ee59b4bd07af27f88d500fcf94325128"
server_key_iv: "884c0c7871424557505136cb22518226"
server_key_version: "encryptionkey@0"
I20260928 16:54:53.546304 27704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:54:53.563802 27704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:54:53.564791 27704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:54:53.565088 27704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:54:53.566216 27704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:54:53.568007 27704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:54:53.568114 27704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:53.568236 27704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:54:53.568311 27704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:53.604068 27704 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:42679
I20260928 16:54:53.604107 27830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:42679 every 8 connection(s)
I20260928 16:54:53.605599 27704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:54:53.611651 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 27704
I20260928 16:54:53.611877 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:54:53.612488 9588 external_mini_cluster.cc:1428] Setting key c4739e972d850dd2a77a25e5be187b02
I20260928 16:54:53.613704 27569 heartbeater.cc:503] Master 127.9.93.62:46751 was elected leader, sending a full tablet report...
I20260928 16:54:53.620769 27702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:53.608803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:46379 (local address 127.9.93.62:46751)
0928 16:54:53.609011 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:53.609016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:53.610737 (+ 1721us) server_negotiation.cc:407] Connection header received
0928 16:54:53.613371 (+ 2634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:53.613378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:53.613480 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:53.613576 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:53.615189 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.616150 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.617851 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.618265 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.618380 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:53.619899 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:53.619911 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:53.619915 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:54:53.619937 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:53.619974 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:54:53.619987 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:53.620065 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:53.620307 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 16:54:53.620449 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 16:54:53.622037 27833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:53.609435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:46751 (local address 127.9.93.3:46379)
0928 16:54:53.610536 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:53.610587 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:53.613008 (+ 2421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:53.613913 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:53.613928 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:53.614080 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:54:53.615017 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:53.615040 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.616318 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.616323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:53.617688 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:53.617703 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.618516 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.618524 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:53.618777 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:53.619537 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:54:53.619540 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:54:53.619588 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:54:53.619683 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:54:53.620083 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:54:53.620095 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:54:53.620104 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 16:54:53.620284 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 16:54:53.620794 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":331,"threads_started":1}
I20260928 16:54:53.623692 27831 heartbeater.cc:347] Connected to a master server at 127.9.93.62:46751
I20260928 16:54:53.624009 27831 heartbeater.cc:464] Registering TS with master...
I20260928 16:54:53.624950 27831 heartbeater.cc:511] Master 127.9.93.62:46751 requested a full tablet report, sending...
I20260928 16:54:53.626648 27382 ts_manager.cc:194] Registered new tserver with Master: a0047951409c483b95417ad48c87d8a6 (127.9.93.3:42679)
I20260928 16:54:53.627820 27382 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.93.3:46379
I20260928 16:54:53.628912 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:54:53.640707 27702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:53.633555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57940 (local address 127.9.93.62:46751)
0928 16:54:53.633849 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:53.633858 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:53.634296 (+ 438us) server_negotiation.cc:407] Connection header received
0928 16:54:53.634478 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:53.634482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:53.634556 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:53.634695 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:53.635387 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.636231 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.637998 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.638490 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.638607 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:53.639852 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:53.639858 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:53.639862 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:54:53.639890 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:53.639929 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:54:53.639943 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:53.640018 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:53.640218 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 16:54:53.640320 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 16:54:53.645612 27382 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:54:53.653196 27382 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:54:53.689939 27843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:53.668125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42956 (local address 127.9.93.2:42623)
0928 16:54:53.680068 (+ 11943us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:53.680079 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:53.680115 (+ 36us) server_negotiation.cc:407] Connection header received
0928 16:54:53.680231 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:53.680240 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:53.680601 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:53.680837 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:54:53.682175 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.683454 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.686845 (+ 3391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.688416 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.688554 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:53.688833 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:53.689334 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 16:54:53.689561 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11736,"thread_start_us":11586,"threads_started":1}
I20260928 16:54:53.690490 27846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:53.668735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37080 (local address 127.9.93.1:37553)
0928 16:54:53.669460 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:53.669471 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:53.679439 (+ 9968us) server_negotiation.cc:407] Connection header received
0928 16:54:53.680697 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:53.680708 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:53.681058 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:53.681269 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:54:53.682278 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.683447 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.686458 (+ 3011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.688054 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.688214 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:53.688595 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:53.689495 (+ 900us) server_negotiation.cc:299] Negotiation successful
0928 16:54:53.689707 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":308,"threads_started":1}
I20260928 16:54:53.690919 27847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:53.668103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:42623 (local address 127.0.0.1:42956)
0928 16:54:53.679178 (+ 11075us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:53.679239 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:53.679505 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:53.681216 (+ 1711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:53.681224 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:53.681259 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:54:53.681838 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:53.681859 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.683848 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.683854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:53.686610 (+ 2756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:53.686626 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.688749 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.688756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:53.688979 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:53.689057 (+ 78us) client_negotiation.cc:769] Sending connection context
0928 16:54:53.690249 (+ 1192us) client_negotiation.cc:240] Negotiation successful
0928 16:54:53.690508 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10705,"mutex_wait_us":7,"thread_start_us":14130,"threads_started":1}
I20260928 16:54:53.691332 27845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:53.668243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:37553 (local address 127.0.0.1:37080)
0928 16:54:53.679201 (+ 10958us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:53.679241 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:53.680267 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:53.681457 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:53.681463 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:53.681487 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:54:53.682094 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:53.682111 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.683598 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.683604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:53.686184 (+ 2580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:53.686201 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.689045 (+ 2844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.689051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:53.689247 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:53.689282 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:54:53.690791 (+ 1509us) client_negotiation.cc:240] Negotiation successful
0928 16:54:53.690944 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10801,"mutex_wait_us":186,"thread_start_us":10655,"threads_started":1}
I20260928 16:54:53.691560 27844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:53.671164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:42679 (local address 127.0.0.1:53496)
0928 16:54:53.682063 (+ 10899us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:53.682083 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:53.683371 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:53.684427 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:53.684433 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:53.684465 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:54:53.684941 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:53.684955 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.687043 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.687049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:53.688508 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:53.688525 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.690532 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.690539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:53.690928 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:53.690962 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 16:54:53.691060 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 16:54:53.691196 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10716,"thread_start_us":7446,"threads_started":1}
I20260928 16:54:53.691829 27848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:53.672205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53496 (local address 127.9.93.3:42679)
0928 16:54:53.681273 (+ 9068us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:53.681279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:53.682229 (+ 950us) server_negotiation.cc:407] Connection header received
0928 16:54:53.683640 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:53.683650 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:53.683983 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:53.684195 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:54:53.685209 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.686896 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.688754 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.690306 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.690464 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:53.690783 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:53.691203 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 16:54:53.691448 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8909,"thread_start_us":8750,"threads_started":1}
I20260928 16:54:53.695873 27504 tablet_service.cc:1467] Processing CreateTablet for tablet 2f7327fa7c27418db2e3c779ba781ec7 (DEFAULT_TABLE table=test-table [id=1927317efd14492b830737db6e2123ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:54:53.695873 27635 tablet_service.cc:1467] Processing CreateTablet for tablet 2f7327fa7c27418db2e3c779ba781ec7 (DEFAULT_TABLE table=test-table [id=1927317efd14492b830737db6e2123ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:54:53.697315 27635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f7327fa7c27418db2e3c779ba781ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:54:53.697327 27504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f7327fa7c27418db2e3c779ba781ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:54:53.701130 27766 tablet_service.cc:1467] Processing CreateTablet for tablet 2f7327fa7c27418db2e3c779ba781ec7 (DEFAULT_TABLE table=test-table [id=1927317efd14492b830737db6e2123ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:54:53.703291 27766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f7327fa7c27418db2e3c779ba781ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:54:53.711512 27849 tablet_bootstrap.cc:492] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35: Bootstrap starting.
I20260928 16:54:53.714274 27850 tablet_bootstrap.cc:492] T 2f7327fa7c27418db2e3c779ba781ec7 P 0f3c9f00f80644ea936051352ba38ca4: Bootstrap starting.
I20260928 16:54:53.717078 27849 tablet_bootstrap.cc:654] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35: Neither blocks nor log segments found. Creating new log.
I20260928 16:54:53.718707 27849 log.cc:824] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35: Log is configured to *not* fsync() on all Append() calls
I20260928 16:54:53.719168 27850 tablet_bootstrap.cc:654] T 2f7327fa7c27418db2e3c779ba781ec7 P 0f3c9f00f80644ea936051352ba38ca4: Neither blocks nor log segments found. Creating new log.
I20260928 16:54:53.720477 27851 tablet_bootstrap.cc:492] T 2f7327fa7c27418db2e3c779ba781ec7 P a0047951409c483b95417ad48c87d8a6: Bootstrap starting.
I20260928 16:54:53.720862 27850 log.cc:824] T 2f7327fa7c27418db2e3c779ba781ec7 P 0f3c9f00f80644ea936051352ba38ca4: Log is configured to *not* fsync() on all Append() calls
I20260928 16:54:53.724658 27850 tablet_bootstrap.cc:492] T 2f7327fa7c27418db2e3c779ba781ec7 P 0f3c9f00f80644ea936051352ba38ca4: No bootstrap required, opened a new log
I20260928 16:54:53.724846 27849 tablet_bootstrap.cc:492] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35: No bootstrap required, opened a new log
I20260928 16:54:53.724983 27850 ts_tablet_manager.cc:1397] T 2f7327fa7c27418db2e3c779ba781ec7 P 0f3c9f00f80644ea936051352ba38ca4: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260928 16:54:53.725031 27851 tablet_bootstrap.cc:654] T 2f7327fa7c27418db2e3c779ba781ec7 P a0047951409c483b95417ad48c87d8a6: Neither blocks nor log segments found. Creating new log.
I20260928 16:54:53.725173 27849 ts_tablet_manager.cc:1397] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260928 16:54:53.726572 27851 log.cc:824] T 2f7327fa7c27418db2e3c779ba781ec7 P a0047951409c483b95417ad48c87d8a6: Log is configured to *not* fsync() on all Append() calls
I20260928 16:54:53.730949 27851 tablet_bootstrap.cc:492] T 2f7327fa7c27418db2e3c779ba781ec7 P a0047951409c483b95417ad48c87d8a6: No bootstrap required, opened a new log
I20260928 16:54:53.731407 27851 ts_tablet_manager.cc:1397] T 2f7327fa7c27418db2e3c779ba781ec7 P a0047951409c483b95417ad48c87d8a6: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.004s
I20260928 16:54:53.733960 27850 raft_consensus.cc:348] T 2f7327fa7c27418db2e3c779ba781ec7 P 0f3c9f00f80644ea936051352ba38ca4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b904ee3a24cc4cc8909b2b381d642d35" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 42623 } } peers { permanent_uuid: "0f3c9f00f80644ea936051352ba38ca4" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 37553 } } peers { permanent_uuid: "a0047951409c483b95417ad48c87d8a6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42679 } }
I20260928 16:54:53.734400 27850 raft_consensus.cc:374] T 2f7327fa7c27418db2e3c779ba781ec7 P 0f3c9f00f80644ea936051352ba38ca4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:54:53.734503 27850 raft_consensus.cc:729] T 2f7327fa7c27418db2e3c779ba781ec7 P 0f3c9f00f80644ea936051352ba38ca4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3c9f00f80644ea936051352ba38ca4, State: Initialized, Role: FOLLOWER
I20260928 16:54:53.735159 27850 consensus_queue.cc:260] T 2f7327fa7c27418db2e3c779ba781ec7 P 0f3c9f00f80644ea936051352ba38ca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b904ee3a24cc4cc8909b2b381d642d35" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 42623 } } peers { permanent_uuid: "0f3c9f00f80644ea936051352ba38ca4" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 37553 } } peers { permanent_uuid: "a0047951409c483b95417ad48c87d8a6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42679 } }
I20260928 16:54:53.735929 27849 raft_consensus.cc:348] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b904ee3a24cc4cc8909b2b381d642d35" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 42623 } } peers { permanent_uuid: "0f3c9f00f80644ea936051352ba38ca4" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 37553 } } peers { permanent_uuid: "a0047951409c483b95417ad48c87d8a6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42679 } }
I20260928 16:54:53.736501 27849 raft_consensus.cc:374] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:54:53.736613 27849 raft_consensus.cc:729] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b904ee3a24cc4cc8909b2b381d642d35, State: Initialized, Role: FOLLOWER
I20260928 16:54:53.737346 27849 consensus_queue.cc:260] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b904ee3a24cc4cc8909b2b381d642d35" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 42623 } } peers { permanent_uuid: "0f3c9f00f80644ea936051352ba38ca4" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 37553 } } peers { permanent_uuid: "a0047951409c483b95417ad48c87d8a6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42679 } }
I20260928 16:54:53.738766 27700 heartbeater.cc:503] Master 127.9.93.62:46751 was elected leader, sending a full tablet report...
I20260928 16:54:53.739812 27849 ts_tablet_manager.cc:1428] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260928 16:54:53.740932 27850 ts_tablet_manager.cc:1428] T 2f7327fa7c27418db2e3c779ba781ec7 P 0f3c9f00f80644ea936051352ba38ca4: Time spent starting tablet: real 0.016s user 0.010s sys 0.007s
I20260928 16:54:53.744331 27851 raft_consensus.cc:348] T 2f7327fa7c27418db2e3c779ba781ec7 P a0047951409c483b95417ad48c87d8a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b904ee3a24cc4cc8909b2b381d642d35" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 42623 } } peers { permanent_uuid: "0f3c9f00f80644ea936051352ba38ca4" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 37553 } } peers { permanent_uuid: "a0047951409c483b95417ad48c87d8a6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42679 } }
I20260928 16:54:53.744944 27851 raft_consensus.cc:374] T 2f7327fa7c27418db2e3c779ba781ec7 P a0047951409c483b95417ad48c87d8a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:54:53.745067 27851 raft_consensus.cc:729] T 2f7327fa7c27418db2e3c779ba781ec7 P a0047951409c483b95417ad48c87d8a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0047951409c483b95417ad48c87d8a6, State: Initialized, Role: FOLLOWER
I20260928 16:54:53.745841 27851 consensus_queue.cc:260] T 2f7327fa7c27418db2e3c779ba781ec7 P a0047951409c483b95417ad48c87d8a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b904ee3a24cc4cc8909b2b381d642d35" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 42623 } } peers { permanent_uuid: "0f3c9f00f80644ea936051352ba38ca4" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 37553 } } peers { permanent_uuid: "a0047951409c483b95417ad48c87d8a6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42679 } }
I20260928 16:54:53.747574 27831 heartbeater.cc:503] Master 127.9.93.62:46751 was elected leader, sending a full tablet report...
I20260928 16:54:53.748111 27851 ts_tablet_manager.cc:1428] T 2f7327fa7c27418db2e3c779ba781ec7 P a0047951409c483b95417ad48c87d8a6: Time spent starting tablet: real 0.017s user 0.013s sys 0.001s
W20260928 16:54:53.835822 27570 tablet.cc:2369] T 2f7327fa7c27418db2e3c779ba781ec7 P 0f3c9f00f80644ea936051352ba38ca4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:54:53.852589 27701 tablet.cc:2369] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:54:53.857542 27832 tablet.cc:2369] T 2f7327fa7c27418db2e3c779ba781ec7 P a0047951409c483b95417ad48c87d8a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:54:53.888023 27856 raft_consensus.cc:482] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:54:53.888406 27856 raft_consensus.cc:504] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b904ee3a24cc4cc8909b2b381d642d35" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 42623 } } peers { permanent_uuid: "0f3c9f00f80644ea936051352ba38ca4" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 37553 } } peers { permanent_uuid: "a0047951409c483b95417ad48c87d8a6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42679 } }
I20260928 16:54:53.890362 27856 leader_election.cc:288] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f3c9f00f80644ea936051352ba38ca4 (127.9.93.1:37553), a0047951409c483b95417ad48c87d8a6 (127.9.93.3:42679)
I20260928 16:54:53.898052 27848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:53.891251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:50279 (local address 127.9.93.3:42679)
0928 16:54:53.891493 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:53.891500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:53.892198 (+ 698us) server_negotiation.cc:407] Connection header received
0928 16:54:53.892397 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:53.892408 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:53.892473 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:53.892547 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:54:53.893190 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.893969 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.896243 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.897074 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.897201 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:53.897307 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:53.897676 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 16:54:53.897809 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:54:53.898043 27846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:53.891216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:42311 (local address 127.9.93.1:37553)
0928 16:54:53.891475 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:53.891483 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:53.892181 (+ 698us) server_negotiation.cc:407] Connection header received
0928 16:54:53.892399 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:53.892403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:53.892484 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:53.892565 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:54:53.893267 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.893990 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.896226 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.897066 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.897188 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:53.897300 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:53.897636 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 16:54:53.897774 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:54:53.898175 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:53.890909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:42679 (local address 127.9.93.2:50279)
0928 16:54:53.892069 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:53.892093 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:53.892255 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:53.892584 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:53.892588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:53.892611 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:54:53.892996 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:53.893007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.894106 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.894112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:53.896014 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:53.896027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.897333 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.897342 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:53.897512 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:53.897536 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:54:53.897625 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 16:54:53.897761 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"mutex_wait_us":143,"spinlock_wait_cycles":29056,"thread_start_us":571,"threads_started":1}
I20260928 16:54:53.898646 27859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:53.890858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:37553 (local address 127.9.93.2:42311)
0928 16:54:53.892030 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:53.892066 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:53.892250 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:53.892598 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:53.892603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:53.892626 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:54:53.893114 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:53.893127 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.894133 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.894137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:53.895998 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:53.896014 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.897296 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.897303 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:53.897455 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:53.897502 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 16:54:53.897576 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 16:54:53.897720 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"mutex_wait_us":116,"spinlock_wait_cycles":1920,"thread_start_us":736,"threads_started":1}
I20260928 16:54:53.899474 27786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7327fa7c27418db2e3c779ba781ec7" candidate_uuid: "b904ee3a24cc4cc8909b2b381d642d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0047951409c483b95417ad48c87d8a6" is_pre_election: true
I20260928 16:54:53.899704 27524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7327fa7c27418db2e3c779ba781ec7" candidate_uuid: "b904ee3a24cc4cc8909b2b381d642d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3c9f00f80644ea936051352ba38ca4" is_pre_election: true
I20260928 16:54:53.900071 27786 raft_consensus.cc:2454] T 2f7327fa7c27418db2e3c779ba781ec7 P a0047951409c483b95417ad48c87d8a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b904ee3a24cc4cc8909b2b381d642d35 in term 0.
I20260928 16:54:53.900252 27524 raft_consensus.cc:2454] T 2f7327fa7c27418db2e3c779ba781ec7 P 0f3c9f00f80644ea936051352ba38ca4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b904ee3a24cc4cc8909b2b381d642d35 in term 0.
I20260928 16:54:53.900951 27590 leader_election.cc:302] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0f3c9f00f80644ea936051352ba38ca4, a0047951409c483b95417ad48c87d8a6, b904ee3a24cc4cc8909b2b381d642d35; no voters:
I20260928 16:54:53.901396 27856 raft_consensus.cc:2781] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:54:53.901584 27856 raft_consensus.cc:482] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:54:53.901693 27856 raft_consensus.cc:3037] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:54:53.904269 27856 raft_consensus.cc:504] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b904ee3a24cc4cc8909b2b381d642d35" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 42623 } } peers { permanent_uuid: "0f3c9f00f80644ea936051352ba38ca4" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 37553 } } peers { permanent_uuid: "a0047951409c483b95417ad48c87d8a6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42679 } }
I20260928 16:54:53.904997 27856 leader_election.cc:288] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35 [CANDIDATE]: Term 1 election: Requested vote from peers 0f3c9f00f80644ea936051352ba38ca4 (127.9.93.1:37553), a0047951409c483b95417ad48c87d8a6 (127.9.93.3:42679)
I20260928 16:54:53.905599 27524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7327fa7c27418db2e3c779ba781ec7" candidate_uuid: "b904ee3a24cc4cc8909b2b381d642d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3c9f00f80644ea936051352ba38ca4"
I20260928 16:54:53.905613 27786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7327fa7c27418db2e3c779ba781ec7" candidate_uuid: "b904ee3a24cc4cc8909b2b381d642d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0047951409c483b95417ad48c87d8a6"
I20260928 16:54:53.905838 27786 raft_consensus.cc:3037] T 2f7327fa7c27418db2e3c779ba781ec7 P a0047951409c483b95417ad48c87d8a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:54:53.905825 27524 raft_consensus.cc:3037] T 2f7327fa7c27418db2e3c779ba781ec7 P 0f3c9f00f80644ea936051352ba38ca4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:54:53.908471 27786 raft_consensus.cc:2454] T 2f7327fa7c27418db2e3c779ba781ec7 P a0047951409c483b95417ad48c87d8a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b904ee3a24cc4cc8909b2b381d642d35 in term 1.
I20260928 16:54:53.908946 27524 raft_consensus.cc:2454] T 2f7327fa7c27418db2e3c779ba781ec7 P 0f3c9f00f80644ea936051352ba38ca4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b904ee3a24cc4cc8909b2b381d642d35 in term 1.
I20260928 16:54:53.909060 27590 leader_election.cc:302] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0047951409c483b95417ad48c87d8a6, b904ee3a24cc4cc8909b2b381d642d35; no voters:
I20260928 16:54:53.909474 27856 raft_consensus.cc:2781] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:54:53.910007 27856 raft_consensus.cc:686] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35 [term 1 LEADER]: Becoming Leader. State: Replica: b904ee3a24cc4cc8909b2b381d642d35, State: Running, Role: LEADER
I20260928 16:54:53.910610 27856 consensus_queue.cc:237] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b904ee3a24cc4cc8909b2b381d642d35" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 42623 } } peers { permanent_uuid: "0f3c9f00f80644ea936051352ba38ca4" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 37553 } } peers { permanent_uuid: "a0047951409c483b95417ad48c87d8a6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42679 } }
I20260928 16:54:53.916119 27381 catalog_manager.cc:5495] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b904ee3a24cc4cc8909b2b381d642d35 (127.9.93.2). New cstate: current_term: 1 leader_uuid: "b904ee3a24cc4cc8909b2b381d642d35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b904ee3a24cc4cc8909b2b381d642d35" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 42623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f3c9f00f80644ea936051352ba38ca4" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 37553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0047951409c483b95417ad48c87d8a6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42679 } health_report { overall_health: UNKNOWN } } }
I20260928 16:54:53.992483 27843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:53.986249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42960 (local address 127.9.93.2:42623)
0928 16:54:53.986483 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:53.986488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:53.986510 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:54:53.986740 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:53.986745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:53.986850 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:53.987005 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:54:53.987440 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.988163 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.989747 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:53.990236 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:53.990366 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:53.991068 (+ 702us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:54:53.991743 (+ 675us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:54:53.991903 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:53.992048 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 16:54:53.992133 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 16:54:54.003106 27786 raft_consensus.cc:1261] T 2f7327fa7c27418db2e3c779ba781ec7 P a0047951409c483b95417ad48c87d8a6 [term 1 FOLLOWER]: Refusing update from remote peer b904ee3a24cc4cc8909b2b381d642d35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:54:54.003106 27524 raft_consensus.cc:1261] T 2f7327fa7c27418db2e3c779ba781ec7 P 0f3c9f00f80644ea936051352ba38ca4 [term 1 FOLLOWER]: Refusing update from remote peer b904ee3a24cc4cc8909b2b381d642d35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:54:54.004592 27860 consensus_queue.cc:1035] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3c9f00f80644ea936051352ba38ca4" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 37553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:54:54.005064 27856 consensus_queue.cc:1035] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0047951409c483b95417ad48c87d8a6" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:54:54.019711 27863 mvcc.cc:205] Tried to move back new op lower bound from 7334356967422402560 to 7334356967062405120. Current Snapshot: MvccSnapshot[applied={T|T < 7334356967422402560}]
I20260928 16:54:54.038731 27382 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:54:54.039005 27382 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:54:54.043606 27382 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84f4641058424c1ab3f37aed110dd210: Sending DeleteTablet for 3 replicas of tablet 2f7327fa7c27418db2e3c779ba781ec7
I20260928 16:54:54.045050 27635 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f7327fa7c27418db2e3c779ba781ec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:54:54 UTC) from {username='slave'} at 127.0.0.1:42956
I20260928 16:54:54.045054 27504 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f7327fa7c27418db2e3c779ba781ec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:54:54 UTC) from {username='slave'} at 127.0.0.1:37080
I20260928 16:54:54.045161 27766 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f7327fa7c27418db2e3c779ba781ec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:54:54 UTC) from {username='slave'} at 127.0.0.1:53496
I20260928 16:54:54.046103 27873 tablet_replica.cc:330] T 2f7327fa7c27418db2e3c779ba781ec7 P a0047951409c483b95417ad48c87d8a6: stopping tablet replica
I20260928 16:54:54.046757 27873 raft_consensus.cc:2229] T 2f7327fa7c27418db2e3c779ba781ec7 P a0047951409c483b95417ad48c87d8a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:54:54.046988 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 27442
I20260928 16:54:54.047747 27873 raft_consensus.cc:2258] T 2f7327fa7c27418db2e3c779ba781ec7 P a0047951409c483b95417ad48c87d8a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:54:54.056046 27875 tablet_replica.cc:330] T 2f7327fa7c27418db2e3c779ba781ec7 P 0f3c9f00f80644ea936051352ba38ca4: stopping tablet replica
I20260928 16:54:54.056164 27873 ts_tablet_manager.cc:1905] T 2f7327fa7c27418db2e3c779ba781ec7 P a0047951409c483b95417ad48c87d8a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:54:54.056628 27875 raft_consensus.cc:2229] T 2f7327fa7c27418db2e3c779ba781ec7 P 0f3c9f00f80644ea936051352ba38ca4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:54:54.057276 27875 raft_consensus.cc:2258] T 2f7327fa7c27418db2e3c779ba781ec7 P 0f3c9f00f80644ea936051352ba38ca4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:54:54.058357 27874 tablet_replica.cc:330] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35: stopping tablet replica
I20260928 16:54:54.058878 27874 raft_consensus.cc:2229] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:54:54.059649 27874 raft_consensus.cc:2258] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:54:54.059849 27875 ts_tablet_manager.cc:1905] T 2f7327fa7c27418db2e3c779ba781ec7 P 0f3c9f00f80644ea936051352ba38ca4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:54:54.062681 27874 ts_tablet_manager.cc:1905] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:54:54.063868 27846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:54.049785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37084 (local address 127.9.93.1:37553)
0928 16:54:54.050085 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:54.050092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:54.050116 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:54:54.050198 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:54.050205 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:54.050348 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:54.050462 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:54.051456 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:54.052871 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:54.055382 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:54.056233 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:54.056358 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:54.059018 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:54.059031 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:54.059050 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:54:54.059931 (+ 881us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:54.060006 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:54:54.060036 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:54.060131 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:54.063257 (+ 3126us) server_negotiation.cc:299] Negotiation successful
0928 16:54:54.063460 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 16:54:54.064369 27873 ts_tablet_manager.cc:1918] T 2f7327fa7c27418db2e3c779ba781ec7 P a0047951409c483b95417ad48c87d8a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:54:54.064582 27873 log.cc:1196] T 2f7327fa7c27418db2e3c779ba781ec7 P a0047951409c483b95417ad48c87d8a6: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/2f7327fa7c27418db2e3c779ba781ec7
I20260928 16:54:54.065862 27564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:54:54.067049 27873 ts_tablet_manager.cc:1939] T 2f7327fa7c27418db2e3c779ba781ec7 P a0047951409c483b95417ad48c87d8a6: Deleting consensus metadata
I20260928 16:54:54.069572 27369 catalog_manager.cc:4841] TS a0047951409c483b95417ad48c87d8a6 (127.9.93.3:42679): tablet 2f7327fa7c27418db2e3c779ba781ec7 (table test-table [id=1927317efd14492b830737db6e2123ed]) successfully deleted
I20260928 16:54:54.074462 27874 ts_tablet_manager.cc:1918] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:54:54.074712 27874 log.cc:1196] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/2f7327fa7c27418db2e3c779ba781ec7
I20260928 16:54:54.075294 27874 ts_tablet_manager.cc:1939] T 2f7327fa7c27418db2e3c779ba781ec7 P b904ee3a24cc4cc8909b2b381d642d35: Deleting consensus metadata
I20260928 16:54:54.077248 27367 catalog_manager.cc:4841] TS b904ee3a24cc4cc8909b2b381d642d35 (127.9.93.2:42623): tablet 2f7327fa7c27418db2e3c779ba781ec7 (table test-table [id=1927317efd14492b830737db6e2123ed]) successfully deleted
I20260928 16:54:54.223357 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 27442
I20260928 16:54:54.247464 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 27573
W20260928 16:54:54.247390 27368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.93.1:37553 (error 108)
I20260928 16:54:54.249925 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:54.249081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:37553 (local address 127.0.0.1:37092)
0928 16:54:54.249532 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:54.249688 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:37553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":226,"threads_started":1}
W20260928 16:54:54.250203 27368 catalog_manager.cc:4584] TS 0f3c9f00f80644ea936051352ba38ca4 (127.9.93.1:37553): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f7327fa7c27418db2e3c779ba781ec7: Network error: Client connection negotiation failed: client connection to 127.9.93.1:37553: connect: Connection refused (error 111)
I20260928 16:54:54.256383 27843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:54.248866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42962 (local address 127.9.93.2:42623)
0928 16:54:54.249096 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:54.249107 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:54.249131 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:54:54.249202 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:54.249207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:54.249330 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:54.249452 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:54.250219 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:54.251286 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:54.252785 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:54.253286 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:54.253429 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:54.254684 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:54.254695 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:54.254712 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:54:54.255506 (+ 794us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:54.255570 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:54:54.255604 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:54.255713 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:54.255883 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 16:54:54.256015 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 16:54:54.257150 27695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:54:54.304198 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:54.303585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:37553 (local address 127.0.0.1:37104)
0928 16:54:54.303858 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:54.303985 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:37553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 16:54:54.305567 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:54.305076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:37553 (local address 127.0.0.1:37116)
0928 16:54:54.305261 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:54.305362 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:37553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 16:54:54.366357 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:54.365748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:37553 (local address 127.0.0.1:37118)
0928 16:54:54.366023 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:54.366144 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:37553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260928 16:54:54.367772 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:54.367253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:37553 (local address 127.0.0.1:37122)
0928 16:54:54.367484 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:54.367583 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:37553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 16:54:54.375748 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 27573
I20260928 16:54:54.393211 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 27704
I20260928 16:54:54.400578 27848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:54.394087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53502 (local address 127.9.93.3:42679)
0928 16:54:54.394279 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:54.394283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:54.394305 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:54:54.394467 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:54.394471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:54.394564 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:54.394661 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:54.395087 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:54.395930 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:54.397397 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:54.397845 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:54.397955 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:54.399102 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:54.399111 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:54.399127 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:54:54.399883 (+ 756us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:54.399947 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:54:54.399971 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:54.400063 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:54.400179 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 16:54:54.400290 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 16:54:54.401481 27826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:54:54.448654 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:54.448125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:37553 (local address 127.0.0.1:37124)
0928 16:54:54.448376 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:54.448486 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:37553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 16:54:54.449874 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:54.449415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:37553 (local address 127.0.0.1:37140)
0928 16:54:54.449642 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:54.449723 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:37553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260928 16:54:54.492949 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 27704
I20260928 16:54:54.508316 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 27352
I20260928 16:54:54.509682 27413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:54:54.626165 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:54.625360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:37553 (local address 127.0.0.1:37146)
0928 16:54:54.625642 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:54.625753 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:37553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260928 16:54:54.626149 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 27352
2026-09-28T16:54:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3249 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:54:54 dist-test-slave-wzcb krb5kdc[27885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:54:54 dist-test-slave-wzcb krb5kdc[27885](info): set up 2 sockets
Sep 28 16:54:54 dist-test-slave-wzcb krb5kdc[27885](info): commencing operation
krb5kdc: starting...
W20260928 16:54:55.686056 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:54:55 dist-test-slave-wzcb krb5kdc[27885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:54:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:54:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:54:55.992686 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:36201
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:46217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:36201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:54:56.386435 27898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:54:56.386825 27898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:54:56.386986 27898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:54:56.394953 27898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:54:56.395273 27898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:54:56.395450 27898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:54:56.395539 27898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:54:56.395586 27898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:54:56.406786 27898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:46217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:36201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:36201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:54:56.408517 27898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:54:56.410175 27898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:54:56.424347 27906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:56.424315 27903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:56.424206 27904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:54:56.425182 27898 server_base.cc:1034] running on GCE node
I20260928 16:54:56.426174 27898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:54:56.428078 27898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:54:56.429368 27898 hybrid_clock.cc:648] HybridClock initialized: now 1790614496429277 us; error 56 us; skew 500 ppm
I20260928 16:54:56.429833 27898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:54:56 dist-test-slave-wzcb krb5kdc[27885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614496, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:54:56.433501 27898 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:54:56.435442 27898 webserver.cc:460] Webserver started at http://127.9.93.62:44437/ using document root <none> and password file <none>
I20260928 16:54:56.436342 27898 fs_manager.cc:359] Metadata directory not provided
I20260928 16:54:56.436478 27898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:54:56.436837 27898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:54:56.440538 27898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "9433f26e16b34e56a1e3eaddf34d6885"
format_stamp: "Formatted at 2026-09-28 16:54:56 on dist-test-slave-wzcb"
server_key: "3b14ee3ac9f62e2d50ef1efc756398c1"
server_key_iv: "a2eeed4aa8af748fde4ca6a0b805c200"
server_key_version: "encryptionkey@0"
I20260928 16:54:56.441519 27898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "9433f26e16b34e56a1e3eaddf34d6885"
format_stamp: "Formatted at 2026-09-28 16:54:56 on dist-test-slave-wzcb"
server_key: "3b14ee3ac9f62e2d50ef1efc756398c1"
server_key_iv: "a2eeed4aa8af748fde4ca6a0b805c200"
server_key_version: "encryptionkey@0"
I20260928 16:54:56.448544 27898 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260928 16:54:56.453466 27913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:56.455569 27898 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.004s
I20260928 16:54:56.455817 27898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "9433f26e16b34e56a1e3eaddf34d6885"
format_stamp: "Formatted at 2026-09-28 16:54:56 on dist-test-slave-wzcb"
server_key: "3b14ee3ac9f62e2d50ef1efc756398c1"
server_key_iv: "a2eeed4aa8af748fde4ca6a0b805c200"
server_key_version: "encryptionkey@0"
I20260928 16:54:56.456081 27898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:54:56.473569 27898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:54:56.478703 27898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:54:56.479048 27898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:54:56.499943 27898 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:36201
I20260928 16:54:56.499975 27964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:36201 every 8 connection(s)
I20260928 16:54:56.501673 27898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:54:56.505455 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 27898
I20260928 16:54:56.505681 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:54:56.506307 9588 external_mini_cluster.cc:1428] Setting key 113ec410e3dc04077ac534d65f49b2eb
I20260928 16:54:56.506775 27965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:54:56.523978 27965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9433f26e16b34e56a1e3eaddf34d6885: Bootstrap starting.
Sep 28 16:54:56 dist-test-slave-wzcb krb5kdc[27885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:54:56.528643 27965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9433f26e16b34e56a1e3eaddf34d6885: Neither blocks nor log segments found. Creating new log.
I20260928 16:54:56.530341 27965 log.cc:824] T 00000000000000000000000000000000 P 9433f26e16b34e56a1e3eaddf34d6885: Log is configured to *not* fsync() on all Append() calls
I20260928 16:54:56.534932 27965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9433f26e16b34e56a1e3eaddf34d6885: No bootstrap required, opened a new log
I20260928 16:54:56.536734 27968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:56.508635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60872 (local address 127.9.93.62:36201)
0928 16:54:56.509538 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:56.509552 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:56.509606 (+ 54us) server_negotiation.cc:407] Connection header received
0928 16:54:56.512413 (+ 2807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:56.512449 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:56.513024 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:56.513469 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:56.515785 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:56.517300 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:56.519515 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:56.520439 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:56.520666 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:56.526182 (+ 5516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:56.526201 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:56.526226 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:56.526301 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:56.530756 (+ 4455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:56.532010 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:56.532021 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:56.532029 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:56.532272 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:56.532808 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:56.532814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:56.532816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:56.533640 (+ 824us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:56.533994 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:56.534499 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 16:54:56.535060 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":382,"threads_started":1}
I20260928 16:54:56.546382 27965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9433f26e16b34e56a1e3eaddf34d6885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9433f26e16b34e56a1e3eaddf34d6885" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 36201 } }
I20260928 16:54:56.546923 27965 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9433f26e16b34e56a1e3eaddf34d6885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:54:56.547055 27965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9433f26e16b34e56a1e3eaddf34d6885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9433f26e16b34e56a1e3eaddf34d6885, State: Initialized, Role: FOLLOWER
I20260928 16:54:56.547798 27965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9433f26e16b34e56a1e3eaddf34d6885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9433f26e16b34e56a1e3eaddf34d6885" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 36201 } }
I20260928 16:54:56.548077 27965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9433f26e16b34e56a1e3eaddf34d6885 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:54:56.548231 27965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9433f26e16b34e56a1e3eaddf34d6885 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:54:56.548414 27965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9433f26e16b34e56a1e3eaddf34d6885 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:54:56.550848 27965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9433f26e16b34e56a1e3eaddf34d6885 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9433f26e16b34e56a1e3eaddf34d6885" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 36201 } }
I20260928 16:54:56.551537 27965 leader_election.cc:302] T 00000000000000000000000000000000 P 9433f26e16b34e56a1e3eaddf34d6885 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9433f26e16b34e56a1e3eaddf34d6885; no voters:
I20260928 16:54:56.552232 27965 leader_election.cc:288] T 00000000000000000000000000000000 P 9433f26e16b34e56a1e3eaddf34d6885 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:54:56.552453 27970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9433f26e16b34e56a1e3eaddf34d6885 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:54:56.553493 27970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9433f26e16b34e56a1e3eaddf34d6885 [term 1 LEADER]: Becoming Leader. State: Replica: 9433f26e16b34e56a1e3eaddf34d6885, State: Running, Role: LEADER
I20260928 16:54:56.554195 27970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9433f26e16b34e56a1e3eaddf34d6885 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9433f26e16b34e56a1e3eaddf34d6885" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 36201 } }
I20260928 16:54:56.554678 27965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9433f26e16b34e56a1e3eaddf34d6885 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:54:56.557330 27971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9433f26e16b34e56a1e3eaddf34d6885 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9433f26e16b34e56a1e3eaddf34d6885" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9433f26e16b34e56a1e3eaddf34d6885" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 36201 } } }
I20260928 16:54:56.557775 27971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9433f26e16b34e56a1e3eaddf34d6885 [sys.catalog]: This master's current role is: LEADER
I20260928 16:54:56.557833 27972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9433f26e16b34e56a1e3eaddf34d6885 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9433f26e16b34e56a1e3eaddf34d6885. Latest consensus state: current_term: 1 leader_uuid: "9433f26e16b34e56a1e3eaddf34d6885" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9433f26e16b34e56a1e3eaddf34d6885" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 36201 } } }
I20260928 16:54:56.558149 27972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9433f26e16b34e56a1e3eaddf34d6885 [sys.catalog]: This master's current role is: LEADER
I20260928 16:54:56.560235 27975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:54:56.568023 27975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:54:56.580484 27975 catalog_manager.cc:1347] Generated new cluster ID: aa7609c65c0e46acb61289c43d2c0077
I20260928 16:54:56.580624 27975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:54:56.589519 27975 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:54:56.591069 27975 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:54:56.605636 27975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9433f26e16b34e56a1e3eaddf34d6885: Generated new TSK 0
I20260928 16:54:56.606881 27975 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
I20260928 16:54:56.699457 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:36201
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:46217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:54:57.085022 27991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:54:57.085372 27991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:54:57.085493 27991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:54:57.093341 27991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:54:57.093495 27991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:54:57.093619 27991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:54:57.104390 27991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:46217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:36201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:54:57.106037 27991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:54:57.107636 27991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:54:57.123342 27996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:57.123288 27999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:57.123386 27997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:54:57.124917 27991 server_base.cc:1034] running on GCE node
I20260928 16:54:57.125726 27991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:54:57.126950 27991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:54:57.128217 27991 hybrid_clock.cc:648] HybridClock initialized: now 1790614497128143 us; error 61 us; skew 500 ppm
I20260928 16:54:57.128636 27991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:54:57 dist-test-slave-wzcb krb5kdc[27885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614497, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:54:57.139606 27991 init.cc:377] Logged in from keytab as kudu/127.9.93.1@KRBTEST.COM (short username kudu)
I20260928 16:54:57.141544 27991 webserver.cc:460] Webserver started at http://127.9.93.1:37021/ using document root <none> and password file <none>
I20260928 16:54:57.142444 27991 fs_manager.cc:359] Metadata directory not provided
I20260928 16:54:57.142594 27991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:54:57.142963 27991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:54:57.146504 27991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "3877665ee3384998a363751ff2813f0a"
format_stamp: "Formatted at 2026-09-28 16:54:57 on dist-test-slave-wzcb"
server_key: "756cbc1a638a6acdc5ea40aeb96666bd"
server_key_iv: "923167ee93b9849007f3504e5c52c5eb"
server_key_version: "encryptionkey@0"
I20260928 16:54:57.147543 27991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "3877665ee3384998a363751ff2813f0a"
format_stamp: "Formatted at 2026-09-28 16:54:57 on dist-test-slave-wzcb"
server_key: "756cbc1a638a6acdc5ea40aeb96666bd"
server_key_iv: "923167ee93b9849007f3504e5c52c5eb"
server_key_version: "encryptionkey@0"
I20260928 16:54:57.154081 27991 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:54:57.158587 28006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:57.160691 27991 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:54:57.160943 27991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "3877665ee3384998a363751ff2813f0a"
format_stamp: "Formatted at 2026-09-28 16:54:57 on dist-test-slave-wzcb"
server_key: "756cbc1a638a6acdc5ea40aeb96666bd"
server_key_iv: "923167ee93b9849007f3504e5c52c5eb"
server_key_version: "encryptionkey@0"
I20260928 16:54:57.161224 27991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:54:57.182152 27991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:54:57.187232 27991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:54:57.187610 27991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:54:57.188777 27991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:54:57.190464 27991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:54:57.190558 27991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:57.190697 27991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:54:57.190755 27991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:57.228448 27991 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:44557
I20260928 16:54:57.228503 28118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:44557 every 8 connection(s)
I20260928 16:54:57.229954 27991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:54:57.231551 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 27991
I20260928 16:54:57.231763 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:54:57.232347 9588 external_mini_cluster.cc:1428] Setting key 5f46963049a040e7efc06a84934c4c97
Sep 28 16:54:57 dist-test-slave-wzcb krb5kdc[27885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614497, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:54:57.253870 28121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:57.233324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:48015 (local address 127.9.93.62:36201)
0928 16:54:57.233852 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:57.233859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:57.235151 (+ 1292us) server_negotiation.cc:407] Connection header received
0928 16:54:57.237715 (+ 2564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:57.237721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:57.237825 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:57.237938 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:57.239980 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:57.240747 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:57.242544 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:57.243001 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:57.243132 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:57.247363 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:57.247372 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:57.247377 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:57.247426 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:57.250139 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:57.251094 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:57.251100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:57.251102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:57.251249 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:57.251813 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:57.251817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:57.251819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:57.252139 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:57.252293 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:57.253208 (+ 915us) server_negotiation.cc:299] Negotiation successful
0928 16:54:57.253494 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":255,"threads_started":1}
I20260928 16:54:57.255136 28122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:57.234060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:36201 (local address 127.9.93.1:48015)
0928 16:54:57.234971 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:57.235024 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:57.237351 (+ 2327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:57.238276 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:57.238286 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:57.238943 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:54:57.239742 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:57.239767 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:57.240938 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:57.240942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:57.242349 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:57.242364 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:57.243288 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:57.243332 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:57.243609 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:57.244335 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:54:57.244368 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:54:57.247014 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:54:57.250349 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:54:57.250359 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:54:57.250376 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:54:57.250928 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:54:57.251418 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:54:57.251424 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:54:57.251428 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:54:57.251676 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:54:57.252285 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:54:57.252295 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:54:57.252751 (+ 456us) client_negotiation.cc:769] Sending connection context
0928 16:54:57.253042 (+ 291us) client_negotiation.cc:240] Negotiation successful
0928 16:54:57.253707 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":364,"threads_started":1}
I20260928 16:54:57.257066 28119 heartbeater.cc:347] Connected to a master server at 127.9.93.62:36201
I20260928 16:54:57.257416 28119 heartbeater.cc:464] Registering TS with master...
I20260928 16:54:57.258282 28119 heartbeater.cc:511] Master 127.9.93.62:36201 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.93.2@KRBTEST.COM; defaulting to no policy
I20260928 16:54:57.261034 27930 ts_manager.cc:194] Registered new tserver with Master: 3877665ee3384998a363751ff2813f0a (127.9.93.1:44557)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.2@KRBTEST.COM" created.
I20260928 16:54:57.263010 27930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.1@KRBTEST.COM'} at 127.9.93.1:48015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
I20260928 16:54:57.295614 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:36201
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:46217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:54:57.665565 28125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:54:57.665930 28125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:54:57.666076 28125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:54:57.674023 28125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:54:57.674185 28125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:54:57.674357 28125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:54:57.684845 28125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:46217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:36201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:54:57.686434 28125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:54:57.688108 28125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:54:57.703298 28130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:57.703437 28131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:57.703373 28133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:54:57.704151 28125 server_base.cc:1034] running on GCE node
I20260928 16:54:57.704921 28125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:54:57.706131 28125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:54:57.707461 28125 hybrid_clock.cc:648] HybridClock initialized: now 1790614497707347 us; error 83 us; skew 500 ppm
I20260928 16:54:57.707916 28125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:54:57 dist-test-slave-wzcb krb5kdc[27885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614497, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:54:57.711889 28125 init.cc:377] Logged in from keytab as kudu/127.9.93.2@KRBTEST.COM (short username kudu)
I20260928 16:54:57.713718 28125 webserver.cc:460] Webserver started at http://127.9.93.2:41389/ using document root <none> and password file <none>
I20260928 16:54:57.714589 28125 fs_manager.cc:359] Metadata directory not provided
I20260928 16:54:57.714713 28125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:54:57.715031 28125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:54:57.718538 28125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "ab7ae3d6a0e340d6aea615baf3b344c7"
format_stamp: "Formatted at 2026-09-28 16:54:57 on dist-test-slave-wzcb"
server_key: "14ac2c7acf6263941d96eba52f39ff0b"
server_key_iv: "2db99d219981d060131ac6932eebc4ee"
server_key_version: "encryptionkey@0"
I20260928 16:54:57.719521 28125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "ab7ae3d6a0e340d6aea615baf3b344c7"
format_stamp: "Formatted at 2026-09-28 16:54:57 on dist-test-slave-wzcb"
server_key: "14ac2c7acf6263941d96eba52f39ff0b"
server_key_iv: "2db99d219981d060131ac6932eebc4ee"
server_key_version: "encryptionkey@0"
I20260928 16:54:57.725971 28125 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:54:57.730468 28140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:57.732376 28125 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:54:57.732596 28125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "ab7ae3d6a0e340d6aea615baf3b344c7"
format_stamp: "Formatted at 2026-09-28 16:54:57 on dist-test-slave-wzcb"
server_key: "14ac2c7acf6263941d96eba52f39ff0b"
server_key_iv: "2db99d219981d060131ac6932eebc4ee"
server_key_version: "encryptionkey@0"
I20260928 16:54:57.732843 28125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:54:57.748461 28125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:54:57.753587 28125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:54:57.753901 28125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:54:57.755159 28125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:54:57.756850 28125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:54:57.756937 28125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:57.757033 28125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:54:57.757081 28125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:57.792317 28125 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:39703
I20260928 16:54:57.792341 28252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:39703 every 8 connection(s)
I20260928 16:54:57.793841 28125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:54:57.796248 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 28125
I20260928 16:54:57.796440 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:54:57.796958 9588 external_mini_cluster.cc:1428] Setting key 3e860650e54849be37bcc18f0513d521
Sep 28 16:54:57 dist-test-slave-wzcb krb5kdc[27885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614497, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:54:57.819638 28255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:57.797022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:57711 (local address 127.9.93.62:36201)
0928 16:54:57.797561 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:57.797567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:57.801162 (+ 3595us) server_negotiation.cc:407] Connection header received
0928 16:54:57.801228 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:57.801232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:57.801310 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:57.801396 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:57.803883 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:57.804724 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:57.807293 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:57.807806 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:57.807934 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:57.812957 (+ 5023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:57.812964 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:57.812967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:57.813012 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:57.815807 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:57.816801 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:57.816806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:57.816809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:57.816979 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:57.817533 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:57.817537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:57.817539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:57.817854 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:57.818021 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:57.818918 (+ 897us) server_negotiation.cc:299] Negotiation successful
0928 16:54:57.819205 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":261,"threads_started":1}
I20260928 16:54:57.820838 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:57.797743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:36201 (local address 127.9.93.2:57711)
0928 16:54:57.798809 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:57.798869 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:57.800817 (+ 1948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:57.801782 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:57.801796 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:57.802657 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:54:57.803643 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:57.803677 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:57.804936 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:57.804941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:57.807068 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:57.807093 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:57.808091 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:57.808099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:57.808443 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:57.809278 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:54:57.809319 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:54:57.812595 (+ 3276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:54:57.816033 (+ 3438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:54:57.816043 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:54:57.816063 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:54:57.816630 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:54:57.817115 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:54:57.817121 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:54:57.817124 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:54:57.817378 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:54:57.818018 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:54:57.818028 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:54:57.818471 (+ 443us) client_negotiation.cc:769] Sending connection context
0928 16:54:57.818749 (+ 278us) client_negotiation.cc:240] Negotiation successful
0928 16:54:57.819442 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":391,"threads_started":1}
I20260928 16:54:57.822757 28253 heartbeater.cc:347] Connected to a master server at 127.9.93.62:36201
I20260928 16:54:57.823100 28253 heartbeater.cc:464] Registering TS with master...
I20260928 16:54:57.824005 28253 heartbeater.cc:511] Master 127.9.93.62:36201 requested a full tablet report, sending...
I20260928 16:54:57.825837 27930 ts_manager.cc:194] Registered new tserver with Master: ab7ae3d6a0e340d6aea615baf3b344c7 (127.9.93.2:39703)
WARNING: no policy specified for kudu/127.9.93.3@KRBTEST.COM; defaulting to no policy
I20260928 16:54:57.826768 27930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.2@KRBTEST.COM'} at 127.9.93.2:57711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
I20260928 16:54:57.864187 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:36201
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:46217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:54:58.232712 28259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:54:58.233162 28259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:54:58.233322 28259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:54:58.242380 28259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:54:58.242527 28259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:54:58.242658 28259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:54:58.253592 28259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:46217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:36201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:54:58.255199 28259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:54:58.256866 28259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:54:58.267045 28119 heartbeater.cc:503] Master 127.9.93.62:36201 was elected leader, sending a full tablet report...
W20260928 16:54:58.272383 28265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:58.272475 28264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:58.272388 28267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:54:58.273614 28259 server_base.cc:1034] running on GCE node
I20260928 16:54:58.274343 28259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:54:58.275497 28259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:54:58.276724 28259 hybrid_clock.cc:648] HybridClock initialized: now 1790614498276658 us; error 60 us; skew 500 ppm
I20260928 16:54:58.277104 28259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:54:58 dist-test-slave-wzcb krb5kdc[27885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614498, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:54:58.280648 28259 init.cc:377] Logged in from keytab as kudu/127.9.93.3@KRBTEST.COM (short username kudu)
I20260928 16:54:58.282487 28259 webserver.cc:460] Webserver started at http://127.9.93.3:33241/ using document root <none> and password file <none>
I20260928 16:54:58.283434 28259 fs_manager.cc:359] Metadata directory not provided
I20260928 16:54:58.283576 28259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:54:58.283928 28259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:54:58.287495 28259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "4225dcb8122d4f48aad739b2fe9128f3"
format_stamp: "Formatted at 2026-09-28 16:54:58 on dist-test-slave-wzcb"
server_key: "3b440b8e6caab84505c1db187d0491f8"
server_key_iv: "95dfa8878717fdc4d654f956ea57c68b"
server_key_version: "encryptionkey@0"
I20260928 16:54:58.288466 28259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "4225dcb8122d4f48aad739b2fe9128f3"
format_stamp: "Formatted at 2026-09-28 16:54:58 on dist-test-slave-wzcb"
server_key: "3b440b8e6caab84505c1db187d0491f8"
server_key_iv: "95dfa8878717fdc4d654f956ea57c68b"
server_key_version: "encryptionkey@0"
I20260928 16:54:58.294816 28259 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 16:54:58.299487 28274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:58.301245 28259 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:54:58.301460 28259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "4225dcb8122d4f48aad739b2fe9128f3"
format_stamp: "Formatted at 2026-09-28 16:54:58 on dist-test-slave-wzcb"
server_key: "3b440b8e6caab84505c1db187d0491f8"
server_key_iv: "95dfa8878717fdc4d654f956ea57c68b"
server_key_version: "encryptionkey@0"
I20260928 16:54:58.301697 28259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:54:58.333194 28259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:54:58.338406 28259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:54:58.338744 28259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:54:58.339980 28259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:54:58.341538 28259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:54:58.341630 28259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:58.341764 28259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:54:58.341828 28259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:58.375279 28259 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:36053
I20260928 16:54:58.375393 28386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:36053 every 8 connection(s)
I20260928 16:54:58.376781 28259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:54:58.385582 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 28259
I20260928 16:54:58.385780 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:54:58.386255 9588 external_mini_cluster.cc:1428] Setting key 116e21a44680926f2febf132572ebbd2
Sep 28 16:54:58 dist-test-slave-wzcb krb5kdc[27885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614498, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:54:58.399587 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:58.379763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:43643 (local address 127.9.93.62:36201)
0928 16:54:58.380259 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:58.380264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:58.381361 (+ 1097us) server_negotiation.cc:407] Connection header received
0928 16:54:58.383450 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:58.383456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:58.383557 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:58.383661 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:58.385634 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.386591 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.388744 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.389210 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.389376 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:58.393310 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:58.393323 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:58.393327 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:58.393372 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:58.396195 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:58.397082 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:58.397086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:58.397088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:58.397225 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:58.397728 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:58.397733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:58.397735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:58.398060 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:58.398190 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:58.398975 (+ 785us) server_negotiation.cc:299] Negotiation successful
0928 16:54:58.399192 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":249,"threads_started":1}
I20260928 16:54:58.400816 28390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:58.380351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:36201 (local address 127.9.93.3:43643)
0928 16:54:58.381168 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:58.381214 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:58.383087 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:58.383963 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:58.383974 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:58.384604 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:54:58.385376 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:58.385422 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.386773 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.386778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:58.388537 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:58.388555 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.389504 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.389511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:58.389818 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:58.390511 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:54:58.390538 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:54:58.393002 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:54:58.396404 (+ 3402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:54:58.396414 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:54:58.396433 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:54:58.396944 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:54:58.397322 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:54:58.397326 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:54:58.397328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:54:58.397572 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:54:58.398169 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:54:58.398176 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:54:58.398596 (+ 420us) client_negotiation.cc:769] Sending connection context
0928 16:54:58.398859 (+ 263us) client_negotiation.cc:240] Negotiation successful
0928 16:54:58.399365 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":321,"threads_started":1}
I20260928 16:54:58.402352 28387 heartbeater.cc:347] Connected to a master server at 127.9.93.62:36201
I20260928 16:54:58.402714 28387 heartbeater.cc:464] Registering TS with master...
I20260928 16:54:58.403569 28387 heartbeater.cc:511] Master 127.9.93.62:36201 requested a full tablet report, sending...
I20260928 16:54:58.405211 27930 ts_manager.cc:194] Registered new tserver with Master: 4225dcb8122d4f48aad739b2fe9128f3 (127.9.93.3:36053)
I20260928 16:54:58.406085 27930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.3@KRBTEST.COM'} at 127.9.93.3:43643
I20260928 16:54:58.415436 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:54:58.435092 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:58.419923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60882 (local address 127.9.93.62:36201)
0928 16:54:58.420167 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:58.420175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:58.420555 (+ 380us) server_negotiation.cc:407] Connection header received
0928 16:54:58.420712 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:58.420716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:58.420802 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:58.420894 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:58.422440 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.423225 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.424790 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.425221 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.425354 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:58.428668 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:58.428675 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:58.428679 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:58.428727 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:58.431625 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:58.432701 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:58.432706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:58.432709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:58.432868 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:58.433421 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:58.433425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:58.433428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:58.433804 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:58.434083 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:58.434548 (+ 465us) server_negotiation.cc:299] Negotiation successful
0928 16:54:58.434780 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 16:54:58.440033 27930 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:54:58.447597 27930 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:54:58.488165 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:58.462218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:36053 (local address 127.0.0.1:43452)
0928 16:54:58.463291 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:58.463369 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:58.463621 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:58.477591 (+ 13970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:58.477604 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:58.477668 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:54:58.478327 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:58.478347 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.481075 (+ 2728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.481083 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:58.484152 (+ 3069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:58.484189 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.486989 (+ 2800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.486998 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:58.487254 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:58.487344 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 16:54:58.487494 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 16:54:58.487757 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"threads_started":1}
I20260928 16:54:58.488616 28402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:58.462569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43452 (local address 127.9.93.3:36053)
0928 16:54:58.476519 (+ 13950us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:58.476532 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:58.476586 (+ 54us) server_negotiation.cc:407] Connection header received
0928 16:54:58.476747 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:58.476758 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:58.477133 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:58.477465 (+ 332us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:54:58.478690 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.480790 (+ 2100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.484566 (+ 3776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.486578 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.487453 (+ 875us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:58.487754 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:58.487852 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 16:54:58.488046 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13710,"thread_start_us":13518,"threads_started":1}
I20260928 16:54:58.496126 28404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:58.462489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:44557 (local address 127.0.0.1:45464)
0928 16:54:58.487247 (+ 24758us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:58.487273 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:58.487495 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:58.488879 (+ 1384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:58.488886 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:58.488919 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:54:58.489359 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:58.489372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.491552 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.491558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:58.493346 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:58.493361 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.495062 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.495069 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:58.495257 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:58.495292 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:54:58.495525 (+ 233us) client_negotiation.cc:240] Negotiation successful
0928 16:54:58.495678 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24589,"mutex_wait_us":106,"thread_start_us":881,"threads_started":1}
I20260928 16:54:58.496380 28403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:58.463152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45464 (local address 127.9.93.1:44557)
0928 16:54:58.486466 (+ 23314us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:58.486478 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:58.487652 (+ 1174us) server_negotiation.cc:407] Connection header received
0928 16:54:58.487819 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:58.487830 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:58.488183 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:58.488444 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:54:58.489575 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.491364 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.493573 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.494838 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.494976 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:58.495223 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:58.495863 (+ 640us) server_negotiation.cc:299] Negotiation successful
0928 16:54:58.496073 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23063,"thread_start_us":23327,"threads_started":1}
I20260928 16:54:58.495651 28322 tablet_service.cc:1467] Processing CreateTablet for tablet ba369e9767614b6997f037abbac93574 (DEFAULT_TABLE table=test-table [id=e65e15478582492fb904e2d49879b770]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:54:58.497247 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:58.467797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:39703 (local address 127.0.0.1:48000)
0928 16:54:58.488547 (+ 20750us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:58.488570 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:58.488700 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:58.489875 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:58.489879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:58.489908 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:54:58.490161 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:58.490175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.492134 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.492139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:58.494026 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:58.494042 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.496474 (+ 2432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.496480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:58.496679 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:58.496714 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:54:58.496813 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 16:54:58.496977 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20630,"thread_start_us":19065,"threads_started":1}
I20260928 16:54:58.497416 28405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:58.468501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48000 (local address 127.9.93.2:39703)
0928 16:54:58.485601 (+ 17100us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:58.485613 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:58.488872 (+ 3259us) server_negotiation.cc:407] Connection header received
0928 16:54:58.489120 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:58.489127 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:58.489345 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:58.489542 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:54:58.490355 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.491962 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.494540 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.496039 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.496175 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:58.496359 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:58.496902 (+ 543us) server_negotiation.cc:299] Negotiation successful
0928 16:54:58.497089 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16822,"thread_start_us":17123,"threads_started":1}
I20260928 16:54:58.497601 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba369e9767614b6997f037abbac93574. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:54:58.500949 28054 tablet_service.cc:1467] Processing CreateTablet for tablet ba369e9767614b6997f037abbac93574 (DEFAULT_TABLE table=test-table [id=e65e15478582492fb904e2d49879b770]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:54:58.501781 28188 tablet_service.cc:1467] Processing CreateTablet for tablet ba369e9767614b6997f037abbac93574 (DEFAULT_TABLE table=test-table [id=e65e15478582492fb904e2d49879b770]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:54:58.502681 28054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba369e9767614b6997f037abbac93574. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:54:58.503500 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba369e9767614b6997f037abbac93574. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:54:58.525081 28407 tablet_bootstrap.cc:492] T ba369e9767614b6997f037abbac93574 P ab7ae3d6a0e340d6aea615baf3b344c7: Bootstrap starting.
I20260928 16:54:58.528255 28406 tablet_bootstrap.cc:492] T ba369e9767614b6997f037abbac93574 P 4225dcb8122d4f48aad739b2fe9128f3: Bootstrap starting.
I20260928 16:54:58.529253 28408 tablet_bootstrap.cc:492] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a: Bootstrap starting.
I20260928 16:54:58.530058 28407 tablet_bootstrap.cc:654] T ba369e9767614b6997f037abbac93574 P ab7ae3d6a0e340d6aea615baf3b344c7: Neither blocks nor log segments found. Creating new log.
I20260928 16:54:58.531338 28407 log.cc:824] T ba369e9767614b6997f037abbac93574 P ab7ae3d6a0e340d6aea615baf3b344c7: Log is configured to *not* fsync() on all Append() calls
I20260928 16:54:58.533079 28406 tablet_bootstrap.cc:654] T ba369e9767614b6997f037abbac93574 P 4225dcb8122d4f48aad739b2fe9128f3: Neither blocks nor log segments found. Creating new log.
I20260928 16:54:58.534339 28406 log.cc:824] T ba369e9767614b6997f037abbac93574 P 4225dcb8122d4f48aad739b2fe9128f3: Log is configured to *not* fsync() on all Append() calls
I20260928 16:54:58.536285 28408 tablet_bootstrap.cc:654] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a: Neither blocks nor log segments found. Creating new log.
I20260928 16:54:58.537833 28408 log.cc:824] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a: Log is configured to *not* fsync() on all Append() calls
I20260928 16:54:58.538056 28407 tablet_bootstrap.cc:492] T ba369e9767614b6997f037abbac93574 P ab7ae3d6a0e340d6aea615baf3b344c7: No bootstrap required, opened a new log
I20260928 16:54:58.538146 28406 tablet_bootstrap.cc:492] T ba369e9767614b6997f037abbac93574 P 4225dcb8122d4f48aad739b2fe9128f3: No bootstrap required, opened a new log
I20260928 16:54:58.538376 28407 ts_tablet_manager.cc:1397] T ba369e9767614b6997f037abbac93574 P ab7ae3d6a0e340d6aea615baf3b344c7: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20260928 16:54:58.538441 28406 ts_tablet_manager.cc:1397] T ba369e9767614b6997f037abbac93574 P 4225dcb8122d4f48aad739b2fe9128f3: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260928 16:54:58.541918 28408 tablet_bootstrap.cc:492] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a: No bootstrap required, opened a new log
I20260928 16:54:58.542423 28408 ts_tablet_manager.cc:1397] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260928 16:54:58.547129 28407 raft_consensus.cc:348] T ba369e9767614b6997f037abbac93574 P ab7ae3d6a0e340d6aea615baf3b344c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4225dcb8122d4f48aad739b2fe9128f3" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36053 } } peers { permanent_uuid: "3877665ee3384998a363751ff2813f0a" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44557 } } peers { permanent_uuid: "ab7ae3d6a0e340d6aea615baf3b344c7" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 39703 } }
I20260928 16:54:58.547672 28407 raft_consensus.cc:374] T ba369e9767614b6997f037abbac93574 P ab7ae3d6a0e340d6aea615baf3b344c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:54:58.547813 28407 raft_consensus.cc:729] T ba369e9767614b6997f037abbac93574 P ab7ae3d6a0e340d6aea615baf3b344c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab7ae3d6a0e340d6aea615baf3b344c7, State: Initialized, Role: FOLLOWER
I20260928 16:54:58.548516 28407 consensus_queue.cc:260] T ba369e9767614b6997f037abbac93574 P ab7ae3d6a0e340d6aea615baf3b344c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4225dcb8122d4f48aad739b2fe9128f3" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36053 } } peers { permanent_uuid: "3877665ee3384998a363751ff2813f0a" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44557 } } peers { permanent_uuid: "ab7ae3d6a0e340d6aea615baf3b344c7" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 39703 } }
I20260928 16:54:58.549793 28253 heartbeater.cc:503] Master 127.9.93.62:36201 was elected leader, sending a full tablet report...
I20260928 16:54:58.550374 28407 ts_tablet_manager.cc:1428] T ba369e9767614b6997f037abbac93574 P ab7ae3d6a0e340d6aea615baf3b344c7: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20260928 16:54:58.549439 28406 raft_consensus.cc:348] T ba369e9767614b6997f037abbac93574 P 4225dcb8122d4f48aad739b2fe9128f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4225dcb8122d4f48aad739b2fe9128f3" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36053 } } peers { permanent_uuid: "3877665ee3384998a363751ff2813f0a" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44557 } } peers { permanent_uuid: "ab7ae3d6a0e340d6aea615baf3b344c7" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 39703 } }
I20260928 16:54:58.550980 28406 raft_consensus.cc:374] T ba369e9767614b6997f037abbac93574 P 4225dcb8122d4f48aad739b2fe9128f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:54:58.551102 28406 raft_consensus.cc:729] T ba369e9767614b6997f037abbac93574 P 4225dcb8122d4f48aad739b2fe9128f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4225dcb8122d4f48aad739b2fe9128f3, State: Initialized, Role: FOLLOWER
I20260928 16:54:58.551841 28406 consensus_queue.cc:260] T ba369e9767614b6997f037abbac93574 P 4225dcb8122d4f48aad739b2fe9128f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4225dcb8122d4f48aad739b2fe9128f3" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36053 } } peers { permanent_uuid: "3877665ee3384998a363751ff2813f0a" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44557 } } peers { permanent_uuid: "ab7ae3d6a0e340d6aea615baf3b344c7" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 39703 } }
I20260928 16:54:58.553248 28387 heartbeater.cc:503] Master 127.9.93.62:36201 was elected leader, sending a full tablet report...
I20260928 16:54:58.553750 28408 raft_consensus.cc:348] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4225dcb8122d4f48aad739b2fe9128f3" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36053 } } peers { permanent_uuid: "3877665ee3384998a363751ff2813f0a" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44557 } } peers { permanent_uuid: "ab7ae3d6a0e340d6aea615baf3b344c7" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 39703 } }
I20260928 16:54:58.554265 28408 raft_consensus.cc:374] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:54:58.554375 28408 raft_consensus.cc:729] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3877665ee3384998a363751ff2813f0a, State: Initialized, Role: FOLLOWER
I20260928 16:54:58.555236 28408 consensus_queue.cc:260] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4225dcb8122d4f48aad739b2fe9128f3" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36053 } } peers { permanent_uuid: "3877665ee3384998a363751ff2813f0a" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44557 } } peers { permanent_uuid: "ab7ae3d6a0e340d6aea615baf3b344c7" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 39703 } }
I20260928 16:54:58.557576 28408 ts_tablet_manager.cc:1428] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260928 16:54:58.557776 28406 ts_tablet_manager.cc:1428] T ba369e9767614b6997f037abbac93574 P 4225dcb8122d4f48aad739b2fe9128f3: Time spent starting tablet: real 0.019s user 0.013s sys 0.000s
I20260928 16:54:58.585798 28414 raft_consensus.cc:482] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:54:58.586197 28414 raft_consensus.cc:504] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4225dcb8122d4f48aad739b2fe9128f3" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36053 } } peers { permanent_uuid: "3877665ee3384998a363751ff2813f0a" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44557 } } peers { permanent_uuid: "ab7ae3d6a0e340d6aea615baf3b344c7" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 39703 } }
I20260928 16:54:58.588191 28414 leader_election.cc:288] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4225dcb8122d4f48aad739b2fe9128f3 (127.9.93.3:36053), ab7ae3d6a0e340d6aea615baf3b344c7 (127.9.93.2:39703)
I20260928 16:54:58.595352 28402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:58.588806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:51053 (local address 127.9.93.3:36053)
0928 16:54:58.589059 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:58.589066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:58.589627 (+ 561us) server_negotiation.cc:407] Connection header received
0928 16:54:58.589817 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:58.589823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:58.589922 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:58.589997 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:54:58.590679 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.591486 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.593498 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.594318 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.594443 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:58.594568 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:58.594909 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 16:54:58.595061 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260928 16:54:58.595366 28405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:58.588877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:35417 (local address 127.9.93.2:39703)
0928 16:54:58.589084 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:58.589090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:58.589783 (+ 693us) server_negotiation.cc:407] Connection header received
0928 16:54:58.589943 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:58.589947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:58.590026 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:58.590124 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:54:58.590736 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.591523 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.593498 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.594322 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.594472 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:58.594577 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:58.594944 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 16:54:58.595084 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:54:58.595546 28416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:58.588639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:39703 (local address 127.9.93.1:35417)
0928 16:54:58.589651 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:58.589674 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:58.589810 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:58.590158 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:58.590162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:58.590199 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:54:58.590561 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:58.590571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.591661 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.591664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:58.593319 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:58.593335 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.594637 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.594643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:58.594776 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:58.594801 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 16:54:58.594892 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 16:54:58.595061 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":98,"thread_start_us":621,"threads_started":1}
I20260928 16:54:58.595582 28415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:58.588556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:36053 (local address 127.9.93.1:51053)
0928 16:54:58.589501 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0928 16:54:58.589519 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:54:58.589669 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:54:58.590038 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:54:58.590043 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:54:58.590079 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:54:58.590520 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:58.590532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.591623 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.591628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:58.593318 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:54:58.593335 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.594581 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.594588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:54:58.594737 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:58.594765 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 16:54:58.594842 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 16:54:58.595016 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"mutex_wait_us":31,"thread_start_us":569,"threads_started":1}
I20260928 16:54:58.596863 28342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba369e9767614b6997f037abbac93574" candidate_uuid: "3877665ee3384998a363751ff2813f0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4225dcb8122d4f48aad739b2fe9128f3" is_pre_election: true
I20260928 16:54:58.596858 28208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba369e9767614b6997f037abbac93574" candidate_uuid: "3877665ee3384998a363751ff2813f0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab7ae3d6a0e340d6aea615baf3b344c7" is_pre_election: true
I20260928 16:54:58.597421 28208 raft_consensus.cc:2454] T ba369e9767614b6997f037abbac93574 P ab7ae3d6a0e340d6aea615baf3b344c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3877665ee3384998a363751ff2813f0a in term 0.
I20260928 16:54:58.597419 28342 raft_consensus.cc:2454] T ba369e9767614b6997f037abbac93574 P 4225dcb8122d4f48aad739b2fe9128f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3877665ee3384998a363751ff2813f0a in term 0.
I20260928 16:54:58.598199 28007 leader_election.cc:302] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3877665ee3384998a363751ff2813f0a, ab7ae3d6a0e340d6aea615baf3b344c7; no voters:
I20260928 16:54:58.598651 28414 raft_consensus.cc:2781] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:54:58.598819 28414 raft_consensus.cc:482] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:54:58.598935 28414 raft_consensus.cc:3037] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:54:58.601366 28414 raft_consensus.cc:504] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4225dcb8122d4f48aad739b2fe9128f3" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36053 } } peers { permanent_uuid: "3877665ee3384998a363751ff2813f0a" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44557 } } peers { permanent_uuid: "ab7ae3d6a0e340d6aea615baf3b344c7" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 39703 } }
I20260928 16:54:58.602090 28414 leader_election.cc:288] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a [CANDIDATE]: Term 1 election: Requested vote from peers 4225dcb8122d4f48aad739b2fe9128f3 (127.9.93.3:36053), ab7ae3d6a0e340d6aea615baf3b344c7 (127.9.93.2:39703)
I20260928 16:54:58.602744 28208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba369e9767614b6997f037abbac93574" candidate_uuid: "3877665ee3384998a363751ff2813f0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab7ae3d6a0e340d6aea615baf3b344c7"
I20260928 16:54:58.602768 28342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba369e9767614b6997f037abbac93574" candidate_uuid: "3877665ee3384998a363751ff2813f0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4225dcb8122d4f48aad739b2fe9128f3"
I20260928 16:54:58.602983 28208 raft_consensus.cc:3037] T ba369e9767614b6997f037abbac93574 P ab7ae3d6a0e340d6aea615baf3b344c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:54:58.603004 28342 raft_consensus.cc:3037] T ba369e9767614b6997f037abbac93574 P 4225dcb8122d4f48aad739b2fe9128f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:54:58.605497 28342 raft_consensus.cc:2454] T ba369e9767614b6997f037abbac93574 P 4225dcb8122d4f48aad739b2fe9128f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3877665ee3384998a363751ff2813f0a in term 1.
I20260928 16:54:58.605619 28208 raft_consensus.cc:2454] T ba369e9767614b6997f037abbac93574 P ab7ae3d6a0e340d6aea615baf3b344c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3877665ee3384998a363751ff2813f0a in term 1.
I20260928 16:54:58.606025 28008 leader_election.cc:302] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3877665ee3384998a363751ff2813f0a, 4225dcb8122d4f48aad739b2fe9128f3; no voters:
I20260928 16:54:58.606446 28414 raft_consensus.cc:2781] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:54:58.607085 28414 raft_consensus.cc:686] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a [term 1 LEADER]: Becoming Leader. State: Replica: 3877665ee3384998a363751ff2813f0a, State: Running, Role: LEADER
I20260928 16:54:58.607703 28414 consensus_queue.cc:237] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4225dcb8122d4f48aad739b2fe9128f3" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36053 } } peers { permanent_uuid: "3877665ee3384998a363751ff2813f0a" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44557 } } peers { permanent_uuid: "ab7ae3d6a0e340d6aea615baf3b344c7" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 39703 } }
I20260928 16:54:58.613171 27929 catalog_manager.cc:5495] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3877665ee3384998a363751ff2813f0a (127.9.93.1). New cstate: current_term: 1 leader_uuid: "3877665ee3384998a363751ff2813f0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4225dcb8122d4f48aad739b2fe9128f3" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3877665ee3384998a363751ff2813f0a" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 44557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab7ae3d6a0e340d6aea615baf3b344c7" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 39703 } health_report { overall_health: UNKNOWN } } }
W20260928 16:54:58.628614 28388 tablet.cc:2369] T ba369e9767614b6997f037abbac93574 P 4225dcb8122d4f48aad739b2fe9128f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:54:58.645627 28403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:58.639432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45466 (local address 127.9.93.1:44557)
0928 16:54:58.639664 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:58.639669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:58.639703 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:54:58.639936 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:58.639940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:58.640059 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:58.640199 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:54:58.640782 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.641563 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.643166 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.643674 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.643790 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:58.644362 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:54:58.644963 (+ 601us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:54:58.645111 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:58.645285 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 16:54:58.645338 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 16:54:58.655795 28342 raft_consensus.cc:1261] T ba369e9767614b6997f037abbac93574 P 4225dcb8122d4f48aad739b2fe9128f3 [term 1 FOLLOWER]: Refusing update from remote peer 3877665ee3384998a363751ff2813f0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:54:58.655789 28208 raft_consensus.cc:1261] T ba369e9767614b6997f037abbac93574 P ab7ae3d6a0e340d6aea615baf3b344c7 [term 1 FOLLOWER]: Refusing update from remote peer 3877665ee3384998a363751ff2813f0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:54:58.657042 28417 consensus_queue.cc:1035] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4225dcb8122d4f48aad739b2fe9128f3" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:54:58.657518 28414 consensus_queue.cc:1035] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ae3d6a0e340d6aea615baf3b344c7" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:54:58.678762 28422 mvcc.cc:205] Tried to move back new op lower bound from 7334356986481528832 to 7334356986301554688. Current Snapshot: MvccSnapshot[applied={T|T < 7334356986481528832}]
I20260928 16:54:58.689738 27928 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:54:58.690066 27928 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:54:58.695144 27928 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9433f26e16b34e56a1e3eaddf34d6885: Sending DeleteTablet for 3 replicas of tablet ba369e9767614b6997f037abbac93574
I20260928 16:54:58.696856 28322 tablet_service.cc:1514] Processing DeleteTablet for tablet ba369e9767614b6997f037abbac93574 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:54:58 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:43452
I20260928 16:54:58.697227 28054 tablet_service.cc:1514] Processing DeleteTablet for tablet ba369e9767614b6997f037abbac93574 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:54:58 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:45464
I20260928 16:54:58.697854 28188 tablet_service.cc:1514] Processing DeleteTablet for tablet ba369e9767614b6997f037abbac93574 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:54:58 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:48000
I20260928 16:54:58.698380 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 27991
I20260928 16:54:58.703770 28431 tablet_replica.cc:330] T ba369e9767614b6997f037abbac93574 P ab7ae3d6a0e340d6aea615baf3b344c7: stopping tablet replica
I20260928 16:54:58.704540 28431 raft_consensus.cc:2229] T ba369e9767614b6997f037abbac93574 P ab7ae3d6a0e340d6aea615baf3b344c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:54:58.705391 28431 raft_consensus.cc:2258] T ba369e9767614b6997f037abbac93574 P ab7ae3d6a0e340d6aea615baf3b344c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:54:58.706650 28430 tablet_replica.cc:330] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a: stopping tablet replica
I20260928 16:54:58.707472 28430 raft_consensus.cc:2229] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:54:58.708470 28430 raft_consensus.cc:2258] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:54:58.709226 28431 ts_tablet_manager.cc:1905] T ba369e9767614b6997f037abbac93574 P ab7ae3d6a0e340d6aea615baf3b344c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:54:58.712994 28429 tablet_replica.cc:330] T ba369e9767614b6997f037abbac93574 P 4225dcb8122d4f48aad739b2fe9128f3: stopping tablet replica
I20260928 16:54:58.713658 28429 raft_consensus.cc:2229] T ba369e9767614b6997f037abbac93574 P 4225dcb8122d4f48aad739b2fe9128f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:54:58.713737 28430 ts_tablet_manager.cc:1905] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:54:58.717995 28431 ts_tablet_manager.cc:1918] T ba369e9767614b6997f037abbac93574 P ab7ae3d6a0e340d6aea615baf3b344c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:54:58.718262 28429 raft_consensus.cc:2258] T ba369e9767614b6997f037abbac93574 P 4225dcb8122d4f48aad739b2fe9128f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:54:58.718379 28431 log.cc:1196] T ba369e9767614b6997f037abbac93574 P ab7ae3d6a0e340d6aea615baf3b344c7: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/ba369e9767614b6997f037abbac93574
I20260928 16:54:58.719135 28431 ts_tablet_manager.cc:1939] T ba369e9767614b6997f037abbac93574 P ab7ae3d6a0e340d6aea615baf3b344c7: Deleting consensus metadata
I20260928 16:54:58.721568 27914 catalog_manager.cc:4841] TS ab7ae3d6a0e340d6aea615baf3b344c7 (127.9.93.2:39703): tablet ba369e9767614b6997f037abbac93574 (table test-table [id=e65e15478582492fb904e2d49879b770]) successfully deleted
I20260928 16:54:58.722069 28429 ts_tablet_manager.cc:1905] T ba369e9767614b6997f037abbac93574 P 4225dcb8122d4f48aad739b2fe9128f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:54:58.724296 28430 ts_tablet_manager.cc:1918] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:54:58.724678 28430 log.cc:1196] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-0/wal/wals/ba369e9767614b6997f037abbac93574
I20260928 16:54:58.725489 28430 ts_tablet_manager.cc:1939] T ba369e9767614b6997f037abbac93574 P 3877665ee3384998a363751ff2813f0a: Deleting consensus metadata
Sep 28 16:54:58 dist-test-slave-wzcb krb5kdc[27885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.1@KRBTEST.COM
I20260928 16:54:58.728010 27914 catalog_manager.cc:4841] TS 3877665ee3384998a363751ff2813f0a (127.9.93.1:44557): tablet ba369e9767614b6997f037abbac93574 (table test-table [id=e65e15478582492fb904e2d49879b770]) successfully deleted
I20260928 16:54:58.731982 28429 ts_tablet_manager.cc:1918] T ba369e9767614b6997f037abbac93574 P 4225dcb8122d4f48aad739b2fe9128f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:54:58.732223 28429 log.cc:1196] T ba369e9767614b6997f037abbac93574 P 4225dcb8122d4f48aad739b2fe9128f3: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/ba369e9767614b6997f037abbac93574
I20260928 16:54:58.732898 28429 ts_tablet_manager.cc:1939] T ba369e9767614b6997f037abbac93574 P 4225dcb8122d4f48aad739b2fe9128f3: Deleting consensus metadata
I20260928 16:54:58.735357 27915 catalog_manager.cc:4841] TS 4225dcb8122d4f48aad739b2fe9128f3 (127.9.93.3:36053): tablet ba369e9767614b6997f037abbac93574 (table test-table [id=e65e15478582492fb904e2d49879b770]) successfully deleted
I20260928 16:54:58.735877 28403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:58.700338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45482 (local address 127.9.93.1:44557)
0928 16:54:58.700576 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:58.700583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:58.709587 (+ 9004us) server_negotiation.cc:407] Connection header received
0928 16:54:58.709982 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:58.709989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:58.710108 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:58.710216 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:58.714106 (+ 3890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.715447 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.719354 (+ 3907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.720470 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.720620 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:58.727523 (+ 6903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:58.727540 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:58.727563 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:58.727614 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:58.731570 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:58.732639 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:58.732652 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:58.732655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:58.732881 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:58.733408 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:58.733413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:58.733415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:58.733880 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:58.734208 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:58.734671 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 16:54:58.734953 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":78}
I20260928 16:54:58.736896 28114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:54:58.844802 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 27991
I20260928 16:54:58.863965 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 28125
Sep 28 16:54:58 dist-test-slave-wzcb krb5kdc[27885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.2@KRBTEST.COM
I20260928 16:54:58.881352 28405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:58.864826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48016 (local address 127.9.93.2:39703)
0928 16:54:58.864997 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:58.865002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:58.865022 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:54:58.865238 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:58.865243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:58.865320 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:58.865420 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:58.866933 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.867877 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.869392 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.869799 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.869901 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:58.874643 (+ 4742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:58.874671 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:58.874697 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:58.874750 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:58.878069 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:58.878993 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:58.879005 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:58.879007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:58.879193 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:58.879648 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:58.879652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:58.879654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:58.880128 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:58.880399 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:58.880811 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 16:54:58.881026 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:54:58.882112 28248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:54:58.973867 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 28125
I20260928 16:54:58.989275 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 28259
Sep 28 16:54:58 dist-test-slave-wzcb krb5kdc[27885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.3@KRBTEST.COM
I20260928 16:54:59.006999 28402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:58.990150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43454 (local address 127.9.93.3:36053)
0928 16:54:58.990360 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:58.990365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:58.990385 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:54:58.990552 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:58.990556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:58.990631 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:58.990730 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:58.992238 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.993009 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.994489 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:58.994908 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:58.995017 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:58.999924 (+ 4907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:58.999946 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:58.999964 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:54:59.000013 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:59.003177 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:59.004228 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:59.004240 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:59.004244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:59.004438 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:54:59.005045 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:54:59.005051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:54:59.005055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:54:59.005580 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:59.005867 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:59.006411 (+ 544us) server_negotiation.cc:299] Negotiation successful
0928 16:54:59.006664 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 16:54:59.007911 28382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:54:59.114007 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 28259
I20260928 16:54:59.129271 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 27898
I20260928 16:54:59.130731 27960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:54:59.249890 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 27898
2026-09-28T16:54:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4624 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T16:54:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:54:59Z Disabled control of system clock
I20260928 16:54:59.335683 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:41723
--webserver_interface=127.9.93.62
--webserver_port=0
--builtin_ntp_servers=127.9.93.20:46171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:41723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:54:59.703861 28440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:54:59.704221 28440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:54:59.704327 28440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:54:59.712225 28440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:54:59.712383 28440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:54:59.712422 28440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:54:59.712477 28440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:54:59.712524 28440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:54:59.722653 28440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:46171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:41723
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:41723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:54:59.724175 28440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:54:59.726253 28440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:54:59.740132 28446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:59.740150 28445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:54:59.740207 28448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:54:59.741799 28440 server_base.cc:1034] running on GCE node
I20260928 16:54:59.742821 28440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:54:59.744587 28440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:54:59.745841 28440 hybrid_clock.cc:648] HybridClock initialized: now 1790614499745775 us; error 49 us; skew 500 ppm
I20260928 16:54:59.746354 28440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:54:59.749073 28440 webserver.cc:460] Webserver started at http://127.9.93.62:44025/ using document root <none> and password file <none>
I20260928 16:54:59.749900 28440 fs_manager.cc:359] Metadata directory not provided
I20260928 16:54:59.750005 28440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:54:59.750378 28440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:54:59.753826 28440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "d894570c16bf4261aa93c0e9d598ed5e"
format_stamp: "Formatted at 2026-09-28 16:54:59 on dist-test-slave-wzcb"
server_key: "6597b0ea97659e26c6dec18c5b484171"
server_key_iv: "e555413cd7ec6d4f8791377670733786"
server_key_version: "encryptionkey@0"
I20260928 16:54:59.754772 28440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "d894570c16bf4261aa93c0e9d598ed5e"
format_stamp: "Formatted at 2026-09-28 16:54:59 on dist-test-slave-wzcb"
server_key: "6597b0ea97659e26c6dec18c5b484171"
server_key_iv: "e555413cd7ec6d4f8791377670733786"
server_key_version: "encryptionkey@0"
I20260928 16:54:59.760931 28440 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 16:54:59.765050 28454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:54:59.766709 28440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:54:59.766925 28440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "d894570c16bf4261aa93c0e9d598ed5e"
format_stamp: "Formatted at 2026-09-28 16:54:59 on dist-test-slave-wzcb"
server_key: "6597b0ea97659e26c6dec18c5b484171"
server_key_iv: "e555413cd7ec6d4f8791377670733786"
server_key_version: "encryptionkey@0"
I20260928 16:54:59.767184 28440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:54:59.793850 28440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:54:59.794819 28440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:54:59.795115 28440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:54:59.813802 28440 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:41723
I20260928 16:54:59.813828 28505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:41723 every 8 connection(s)
I20260928 16:54:59.815296 28440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:54:59.815959 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 28440
I20260928 16:54:59.816166 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:54:59.816828 9588 external_mini_cluster.cc:1428] Setting key 4fbd9ac0bd4fb40cecf4eba671626b5b
I20260928 16:54:59.821003 28506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:54:59.832957 28509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:54:59.819243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36446 (local address 127.9.93.62:41723)
0928 16:54:59.820437 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0928 16:54:59.820450 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:54:59.820500 (+ 50us) server_negotiation.cc:407] Connection header received
0928 16:54:59.822335 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:54:59.822360 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:54:59.822876 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:54:59.823265 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:54:59.824105 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:59.825693 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:59.827799 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:54:59.828596 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:54:59.828750 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:54:59.830159 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:54:59.830167 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:54:59.830184 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:54:59.830869 (+ 685us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:54:59.830959 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:54:59.830985 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:54:59.831080 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 16:54:59.831382 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 16:54:59.831813 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":383,"threads_started":1}
I20260928 16:54:59.834782 28506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d894570c16bf4261aa93c0e9d598ed5e: Bootstrap starting.
I20260928 16:54:59.838372 28506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d894570c16bf4261aa93c0e9d598ed5e: Neither blocks nor log segments found. Creating new log.
I20260928 16:54:59.839568 28506 log.cc:824] T 00000000000000000000000000000000 P d894570c16bf4261aa93c0e9d598ed5e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:54:59.842875 28506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d894570c16bf4261aa93c0e9d598ed5e: No bootstrap required, opened a new log
I20260928 16:54:59.850961 28506 raft_consensus.cc:348] T 00000000000000000000000000000000 P d894570c16bf4261aa93c0e9d598ed5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d894570c16bf4261aa93c0e9d598ed5e" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 41723 } }
I20260928 16:54:59.851440 28506 raft_consensus.cc:374] T 00000000000000000000000000000000 P d894570c16bf4261aa93c0e9d598ed5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:54:59.851545 28506 raft_consensus.cc:729] T 00000000000000000000000000000000 P d894570c16bf4261aa93c0e9d598ed5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d894570c16bf4261aa93c0e9d598ed5e, State: Initialized, Role: FOLLOWER
I20260928 16:54:59.852202 28506 consensus_queue.cc:260] T 00000000000000000000000000000000 P d894570c16bf4261aa93c0e9d598ed5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d894570c16bf4261aa93c0e9d598ed5e" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 41723 } }
I20260928 16:54:59.852496 28506 raft_consensus.cc:388] T 00000000000000000000000000000000 P d894570c16bf4261aa93c0e9d598ed5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:54:59.852635 28506 raft_consensus.cc:482] T 00000000000000000000000000000000 P d894570c16bf4261aa93c0e9d598ed5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:54:59.852821 28506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d894570c16bf4261aa93c0e9d598ed5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:54:59.855412 28506 raft_consensus.cc:504] T 00000000000000000000000000000000 P d894570c16bf4261aa93c0e9d598ed5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d894570c16bf4261aa93c0e9d598ed5e" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 41723 } }
I20260928 16:54:59.855990 28506 leader_election.cc:302] T 00000000000000000000000000000000 P d894570c16bf4261aa93c0e9d598ed5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d894570c16bf4261aa93c0e9d598ed5e; no voters:
I20260928 16:54:59.856725 28506 leader_election.cc:288] T 00000000000000000000000000000000 P d894570c16bf4261aa93c0e9d598ed5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:54:59.856827 28511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d894570c16bf4261aa93c0e9d598ed5e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:54:59.857595 28511 raft_consensus.cc:686] T 00000000000000000000000000000000 P d894570c16bf4261aa93c0e9d598ed5e [term 1 LEADER]: Becoming Leader. State: Replica: d894570c16bf4261aa93c0e9d598ed5e, State: Running, Role: LEADER
I20260928 16:54:59.858202 28511 consensus_queue.cc:237] T 00000000000000000000000000000000 P d894570c16bf4261aa93c0e9d598ed5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d894570c16bf4261aa93c0e9d598ed5e" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 41723 } }
I20260928 16:54:59.859032 28506 sys_catalog.cc:564] T 00000000000000000000000000000000 P d894570c16bf4261aa93c0e9d598ed5e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:54:59.862193 28512 sys_catalog.cc:455] T 00000000000000000000000000000000 P d894570c16bf4261aa93c0e9d598ed5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d894570c16bf4261aa93c0e9d598ed5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d894570c16bf4261aa93c0e9d598ed5e" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 41723 } } }
I20260928 16:54:59.862874 28513 sys_catalog.cc:455] T 00000000000000000000000000000000 P d894570c16bf4261aa93c0e9d598ed5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d894570c16bf4261aa93c0e9d598ed5e. Latest consensus state: current_term: 1 leader_uuid: "d894570c16bf4261aa93c0e9d598ed5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d894570c16bf4261aa93c0e9d598ed5e" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 41723 } } }
I20260928 16:54:59.863217 28512 sys_catalog.cc:458] T 00000000000000000000000000000000 P d894570c16bf4261aa93c0e9d598ed5e [sys.catalog]: This master's current role is: LEADER
I20260928 16:54:59.863233 28513 sys_catalog.cc:458] T 00000000000000000000000000000000 P d894570c16bf4261aa93c0e9d598ed5e [sys.catalog]: This master's current role is: LEADER
I20260928 16:54:59.864343 28519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:54:59.870884 28519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:54:59.887223 28519 catalog_manager.cc:1347] Generated new cluster ID: 076271ee35e34eba87307660d796764b
I20260928 16:54:59.887395 28519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:54:59.917585 28519 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:54:59.919078 28519 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:54:59.927402 28519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d894570c16bf4261aa93c0e9d598ed5e: Generated new TSK 0
I20260928 16:54:59.928390 28519 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:54:59.939639 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:41723
--builtin_ntp_servers=127.9.93.20:46171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:55:00.322250 28530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:00.322592 28530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:00.322710 28530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:00.330575 28530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:00.330696 28530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:00.330844 28530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:55:00.341315 28530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:46171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:41723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:00.342856 28530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:00.344992 28530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:00.360287 28536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:00.360476 28535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:00.360260 28538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:00.361763 28530 server_base.cc:1034] running on GCE node
I20260928 16:55:00.362470 28530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:00.363636 28530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:00.364854 28530 hybrid_clock.cc:648] HybridClock initialized: now 1790614500364807 us; error 41 us; skew 500 ppm
I20260928 16:55:00.365247 28530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:55:00.367964 28530 webserver.cc:460] Webserver started at http://127.9.93.1:46579/ using document root <none> and password file <none>
I20260928 16:55:00.368754 28530 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:00.368868 28530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:00.369213 28530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:00.372943 28530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "b5da4d315109496e97d99f4f5422780f"
format_stamp: "Formatted at 2026-09-28 16:55:00 on dist-test-slave-wzcb"
server_key: "cc2ff719d0f5d2c06bf2f289cc52fd68"
server_key_iv: "075057c1c4057b43400fea926d624dec"
server_key_version: "encryptionkey@0"
I20260928 16:55:00.373832 28530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "b5da4d315109496e97d99f4f5422780f"
format_stamp: "Formatted at 2026-09-28 16:55:00 on dist-test-slave-wzcb"
server_key: "cc2ff719d0f5d2c06bf2f289cc52fd68"
server_key_iv: "075057c1c4057b43400fea926d624dec"
server_key_version: "encryptionkey@0"
I20260928 16:55:00.379944 28530 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 16:55:00.384266 28544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:00.385871 28530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:55:00.386098 28530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "b5da4d315109496e97d99f4f5422780f"
format_stamp: "Formatted at 2026-09-28 16:55:00 on dist-test-slave-wzcb"
server_key: "cc2ff719d0f5d2c06bf2f289cc52fd68"
server_key_iv: "075057c1c4057b43400fea926d624dec"
server_key_version: "encryptionkey@0"
I20260928 16:55:00.386351 28530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:00.412847 28530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:00.413837 28530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:00.414144 28530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:00.415261 28530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:55:00.417034 28530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:55:00.417141 28530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:00.417259 28530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:55:00.417333 28530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:00.450207 28530 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:36231
I20260928 16:55:00.450285 28656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:36231 every 8 connection(s)
I20260928 16:55:00.451784 28530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:55:00.453308 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 28530
I20260928 16:55:00.453501 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:55:00.454097 9588 external_mini_cluster.cc:1428] Setting key e605dd33fadff8ea41d8d8a3e678d742
I20260928 16:55:00.457077 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:41723
--builtin_ntp_servers=127.9.93.20:46171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:55:00.467581 28659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:00.454908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:51715 (local address 127.9.93.62:41723)
0928 16:55:00.455470 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:00.455477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:00.457133 (+ 1656us) server_negotiation.cc:407] Connection header received
0928 16:55:00.459376 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:00.459383 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:00.459489 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:00.459616 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:00.461063 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:00.462056 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:00.464227 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:00.464886 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:00.465016 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:00.466657 (+ 1641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:00.466664 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:00.466668 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:00.466694 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:00.466750 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:00.466766 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:00.466854 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:00.467073 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 16:55:00.467189 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":254,"threads_started":1}
I20260928 16:55:00.469067 28660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:00.455706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:41723 (local address 127.9.93.1:51715)
0928 16:55:00.456946 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:00.456998 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:00.459020 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:00.459889 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:00.459901 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:00.460038 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:00.460854 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:00.460877 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:00.462216 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:00.462221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:00.464035 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:00.464051 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:00.465136 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:00.465142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:00.465481 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:00.466255 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:55:00.466258 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:00.466313 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:55:00.466433 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:00.466855 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:00.466863 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:00.466870 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 16:55:00.467056 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 16:55:00.467539 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":382,"threads_started":1}
I20260928 16:55:00.471029 28657 heartbeater.cc:347] Connected to a master server at 127.9.93.62:41723
I20260928 16:55:00.471432 28657 heartbeater.cc:464] Registering TS with master...
I20260928 16:55:00.472363 28657 heartbeater.cc:511] Master 127.9.93.62:41723 requested a full tablet report, sending...
I20260928 16:55:00.474845 28471 ts_manager.cc:194] Registered new tserver with Master: b5da4d315109496e97d99f4f5422780f (127.9.93.1:36231)
I20260928 16:55:00.476567 28471 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.93.1:51715
W20260928 16:55:00.819026 28661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:00.819392 28661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:00.819545 28661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:00.827411 28661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:00.827533 28661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:00.827689 28661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:55:00.837630 28661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:46171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:41723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:00.839129 28661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:00.841279 28661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:00.856043 28666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:00.855976 28667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:00.856012 28669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:00.857193 28661 server_base.cc:1034] running on GCE node
I20260928 16:55:00.858021 28661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:00.859251 28661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:00.860541 28661 hybrid_clock.cc:648] HybridClock initialized: now 1790614500860465 us; error 67 us; skew 500 ppm
I20260928 16:55:00.860915 28661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:55:00.863770 28661 webserver.cc:460] Webserver started at http://127.9.93.2:35803/ using document root <none> and password file <none>
I20260928 16:55:00.864639 28661 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:00.864765 28661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:00.865111 28661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:00.868582 28661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "f3835a835706486f85fd8baa80ce48da"
format_stamp: "Formatted at 2026-09-28 16:55:00 on dist-test-slave-wzcb"
server_key: "5f289a3242f9524f345174715099ef72"
server_key_iv: "038c5fef7c5a49e9a7ffcdbd4196c937"
server_key_version: "encryptionkey@0"
I20260928 16:55:00.869588 28661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "f3835a835706486f85fd8baa80ce48da"
format_stamp: "Formatted at 2026-09-28 16:55:00 on dist-test-slave-wzcb"
server_key: "5f289a3242f9524f345174715099ef72"
server_key_iv: "038c5fef7c5a49e9a7ffcdbd4196c937"
server_key_version: "encryptionkey@0"
I20260928 16:55:00.875906 28661 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 16:55:00.880250 28675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:00.882107 28661 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:55:00.882339 28661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "f3835a835706486f85fd8baa80ce48da"
format_stamp: "Formatted at 2026-09-28 16:55:00 on dist-test-slave-wzcb"
server_key: "5f289a3242f9524f345174715099ef72"
server_key_iv: "038c5fef7c5a49e9a7ffcdbd4196c937"
server_key_version: "encryptionkey@0"
I20260928 16:55:00.882596 28661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:00.904147 28661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:00.905167 28661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:00.905481 28661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:00.906675 28661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:55:00.908525 28661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:55:00.908627 28661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:00.908723 28661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:55:00.908799 28661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:00.941277 28661 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:46717
I20260928 16:55:00.941318 28787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:46717 every 8 connection(s)
I20260928 16:55:00.942814 28661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:55:00.949817 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 28661
I20260928 16:55:00.950047 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:55:00.950536 9588 external_mini_cluster.cc:1428] Setting key 7502b01868d378651e7b5e5b7ab3c558
I20260928 16:55:00.953949 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:41723
--builtin_ntp_servers=127.9.93.20:46171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:55:00.957861 28659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:00.945807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:41175 (local address 127.9.93.62:41723)
0928 16:55:00.946067 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:00.946074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:00.947775 (+ 1701us) server_negotiation.cc:407] Connection header received
0928 16:55:00.949742 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:00.949748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:00.949852 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:00.949996 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:00.951534 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:00.952368 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:00.954482 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:00.954981 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:00.955125 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:00.956855 (+ 1730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:00.956861 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:00.956865 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:00.956891 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:00.956929 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:00.956943 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:00.957034 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:00.957322 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 16:55:00.957458 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 16:55:00.959113 28790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:00.946498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:41723 (local address 127.9.93.2:41175)
0928 16:55:00.947590 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:00.947639 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:00.949395 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:00.950344 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:00.950355 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:00.950466 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:00.951298 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:00.951362 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:00.952602 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:00.952609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:00.954294 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:00.954312 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:00.955276 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:00.955287 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:00.955603 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:00.956422 (+ 819us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:55:00.956425 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:00.956475 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:55:00.956588 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:00.957049 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:00.957064 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:00.957074 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 16:55:00.957274 (+ 200us) client_negotiation.cc:240] Negotiation successful
0928 16:55:00.957785 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":338,"threads_started":1}
I20260928 16:55:00.961122 28788 heartbeater.cc:347] Connected to a master server at 127.9.93.62:41723
I20260928 16:55:00.961534 28788 heartbeater.cc:464] Registering TS with master...
I20260928 16:55:00.962435 28788 heartbeater.cc:511] Master 127.9.93.62:41723 requested a full tablet report, sending...
I20260928 16:55:00.964141 28471 ts_manager.cc:194] Registered new tserver with Master: f3835a835706486f85fd8baa80ce48da (127.9.93.2:46717)
I20260928 16:55:00.965096 28471 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.93.2:41175
W20260928 16:55:01.332880 28791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:01.333266 28791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:01.333410 28791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:01.341171 28791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:01.341312 28791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:01.341466 28791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:55:01.351261 28791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:46171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:41723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:01.352895 28791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:01.354935 28791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:01.370036 28797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:01.370014 28799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:01.369973 28796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:01.370947 28791 server_base.cc:1034] running on GCE node
I20260928 16:55:01.371716 28791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:01.372948 28791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:01.374171 28791 hybrid_clock.cc:648] HybridClock initialized: now 1790614501374116 us; error 45 us; skew 500 ppm
I20260928 16:55:01.374508 28791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:55:01.377013 28791 webserver.cc:460] Webserver started at http://127.9.93.3:46019/ using document root <none> and password file <none>
I20260928 16:55:01.377861 28791 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:01.377967 28791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:01.378268 28791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:01.381968 28791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "9699d6741cbe4c0bb0c2561a6e2cdb68"
format_stamp: "Formatted at 2026-09-28 16:55:01 on dist-test-slave-wzcb"
server_key: "262c2c5de85702b239a789688611536e"
server_key_iv: "b40dae9e9685053afe97d20fdba7c598"
server_key_version: "encryptionkey@0"
I20260928 16:55:01.382819 28791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "9699d6741cbe4c0bb0c2561a6e2cdb68"
format_stamp: "Formatted at 2026-09-28 16:55:01 on dist-test-slave-wzcb"
server_key: "262c2c5de85702b239a789688611536e"
server_key_iv: "b40dae9e9685053afe97d20fdba7c598"
server_key_version: "encryptionkey@0"
I20260928 16:55:01.388975 28791 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 16:55:01.393210 28805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:01.394963 28791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:55:01.395164 28791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "9699d6741cbe4c0bb0c2561a6e2cdb68"
format_stamp: "Formatted at 2026-09-28 16:55:01 on dist-test-slave-wzcb"
server_key: "262c2c5de85702b239a789688611536e"
server_key_iv: "b40dae9e9685053afe97d20fdba7c598"
server_key_version: "encryptionkey@0"
I20260928 16:55:01.395450 28791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:01.412923 28791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:01.413967 28791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:01.414283 28791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:01.415560 28791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:55:01.417330 28791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:55:01.417434 28791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:01.417543 28791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:55:01.417610 28791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:01.450182 28791 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:34563
I20260928 16:55:01.450276 28917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:34563 every 8 connection(s)
I20260928 16:55:01.451720 28791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:55:01.456067 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 28791
I20260928 16:55:01.456285 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:55:01.456914 9588 external_mini_cluster.cc:1428] Setting key 0c060677c27d2898138da342ac3b7944
I20260928 16:55:01.466399 28659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:01.455218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:43585 (local address 127.9.93.62:41723)
0928 16:55:01.455469 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:01.455476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:01.457258 (+ 1782us) server_negotiation.cc:407] Connection header received
0928 16:55:01.459447 (+ 2189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:01.459453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:01.459572 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:01.459668 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:01.461037 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.461801 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.463472 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.463890 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.464022 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:01.465539 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:01.465544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:01.465548 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:01.465570 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:01.465604 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:01.465616 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:01.465693 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:01.465921 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 16:55:01.466043 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260928 16:55:01.467630 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:01.455890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:41723 (local address 127.9.93.3:43585)
0928 16:55:01.457085 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:01.457129 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:01.459123 (+ 1994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:01.459955 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:01.459965 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:01.460108 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:01.460861 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:01.460883 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.461961 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.461966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:01.463286 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:01.463330 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.464134 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.464141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:01.464407 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:01.465157 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:55:01.465160 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:01.465208 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:55:01.465301 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:01.465707 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:01.465719 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:01.465728 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 16:55:01.465900 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 16:55:01.466388 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":416,"threads_started":1}
I20260928 16:55:01.469208 28918 heartbeater.cc:347] Connected to a master server at 127.9.93.62:41723
I20260928 16:55:01.469506 28918 heartbeater.cc:464] Registering TS with master...
I20260928 16:55:01.470369 28918 heartbeater.cc:511] Master 127.9.93.62:41723 requested a full tablet report, sending...
I20260928 16:55:01.472250 28471 ts_manager.cc:194] Registered new tserver with Master: 9699d6741cbe4c0bb0c2561a6e2cdb68 (127.9.93.3:34563)
I20260928 16:55:01.473028 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:55:01.473292 28471 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.93.3:43585
I20260928 16:55:01.480113 28657 heartbeater.cc:503] Master 127.9.93.62:41723 was elected leader, sending a full tablet report...
I20260928 16:55:01.485035 28659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:01.478141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36454 (local address 127.9.93.62:41723)
0928 16:55:01.478350 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:01.478357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:01.478782 (+ 425us) server_negotiation.cc:407] Connection header received
0928 16:55:01.478999 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:01.479006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:01.479095 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:01.479193 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:01.479750 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.480536 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.482406 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.482862 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.482970 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:01.484095 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:01.484100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:01.484103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:01.484125 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:01.484161 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:01.484172 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:01.484334 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:01.484625 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 16:55:01.484700 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 16:55:01.489838 28471 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:55:01.497263 28471 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:55:01.526466 28931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:01.512277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50196 (local address 127.9.93.2:46717)
0928 16:55:01.513004 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:01.513011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:01.513151 (+ 140us) server_negotiation.cc:407] Connection header received
0928 16:55:01.513494 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:01.513502 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:01.513817 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:01.514040 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:55:01.516670 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.518460 (+ 1790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.523228 (+ 4768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.524805 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.524962 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:01.525257 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:01.525883 (+ 626us) server_negotiation.cc:299] Negotiation successful
0928 16:55:01.526095 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":448,"threads_started":1}
I20260928 16:55:01.526507 28930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:01.511994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:46717 (local address 127.0.0.1:50196)
0928 16:55:01.513001 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:01.513050 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:01.513249 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:01.514332 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:01.514340 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:01.514369 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:55:01.516419 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:01.516442 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.520336 (+ 3894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.520343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:01.522990 (+ 2647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:01.523007 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.525113 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.525121 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:01.525391 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:01.525645 (+ 254us) client_negotiation.cc:769] Sending connection context
0928 16:55:01.525826 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 16:55:01.526079 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":129,"thread_start_us":569,"threads_started":1}
I20260928 16:55:01.530380 28932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:01.513494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:36231 (local address 127.0.0.1:35782)
0928 16:55:01.514089 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:01.514118 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:01.514229 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:01.522727 (+ 8498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:01.522732 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:01.522763 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:55:01.523218 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:01.523232 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.525226 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.525231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:01.527402 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:01.527418 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.529488 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.529496 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:01.529692 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:01.529728 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 16:55:01.529799 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 16:55:01.529966 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":3,"thread_start_us":287,"threads_started":1}
I20260928 16:55:01.531073 28935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:01.514833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42132 (local address 127.9.93.3:34563)
0928 16:55:01.521577 (+ 6744us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:01.521589 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:01.521638 (+ 49us) server_negotiation.cc:407] Connection header received
0928 16:55:01.521763 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:01.521773 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:01.522105 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:01.522334 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:55:01.523110 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.524792 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.527804 (+ 3012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.529228 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.529388 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:01.529706 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:01.530493 (+ 787us) server_negotiation.cc:299] Negotiation successful
0928 16:55:01.530694 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6498,"thread_start_us":6483,"threads_started":1}
I20260928 16:55:01.531383 28934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:01.514180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:34563 (local address 127.0.0.1:42132)
0928 16:55:01.521064 (+ 6884us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:01.521087 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:01.521242 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:01.522333 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:01.522337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:01.522367 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:55:01.522827 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:01.522840 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.524972 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.524978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:01.527568 (+ 2590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:01.527584 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.529856 (+ 2272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.529863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:01.530064 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:01.530099 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:55:01.530179 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 16:55:01.530325 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6726,"mutex_wait_us":158,"thread_start_us":6541,"threads_started":1}
I20260928 16:55:01.531491 28933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:01.513818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35782 (local address 127.9.93.1:36231)
0928 16:55:01.522065 (+ 8247us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:01.522075 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:01.522108 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:55:01.522218 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:01.522226 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:01.522504 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:01.522727 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:55:01.523485 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.525061 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.527659 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.529229 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.530537 (+ 1308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:01.530840 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:01.530940 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 16:55:01.531119 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8052,"thread_start_us":7916,"threads_started":1}
I20260928 16:55:01.531416 28723 tablet_service.cc:1467] Processing CreateTablet for tablet d208e27cce54405faa75a6aa0a4c6bd4 (DEFAULT_TABLE table=test-table [id=9de55b390c484b2fbd6e1e9c4eb644e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:55:01.533174 28723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d208e27cce54405faa75a6aa0a4c6bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:01.535898 28592 tablet_service.cc:1467] Processing CreateTablet for tablet d208e27cce54405faa75a6aa0a4c6bd4 (DEFAULT_TABLE table=test-table [id=9de55b390c484b2fbd6e1e9c4eb644e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:55:01.535897 28853 tablet_service.cc:1467] Processing CreateTablet for tablet d208e27cce54405faa75a6aa0a4c6bd4 (DEFAULT_TABLE table=test-table [id=9de55b390c484b2fbd6e1e9c4eb644e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:55:01.537789 28853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d208e27cce54405faa75a6aa0a4c6bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:01.538205 28592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d208e27cce54405faa75a6aa0a4c6bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:01.547377 28936 tablet_bootstrap.cc:492] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da: Bootstrap starting.
I20260928 16:55:01.552587 28936 tablet_bootstrap.cc:654] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:01.556384 28936 log.cc:824] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:01.558333 28938 tablet_bootstrap.cc:492] T d208e27cce54405faa75a6aa0a4c6bd4 P 9699d6741cbe4c0bb0c2561a6e2cdb68: Bootstrap starting.
I20260928 16:55:01.559669 28937 tablet_bootstrap.cc:492] T d208e27cce54405faa75a6aa0a4c6bd4 P b5da4d315109496e97d99f4f5422780f: Bootstrap starting.
I20260928 16:55:01.560760 28936 tablet_bootstrap.cc:492] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da: No bootstrap required, opened a new log
I20260928 16:55:01.561118 28936 ts_tablet_manager.cc:1397] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.006s
I20260928 16:55:01.563963 28938 tablet_bootstrap.cc:654] T d208e27cce54405faa75a6aa0a4c6bd4 P 9699d6741cbe4c0bb0c2561a6e2cdb68: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:01.566164 28938 log.cc:824] T d208e27cce54405faa75a6aa0a4c6bd4 P 9699d6741cbe4c0bb0c2561a6e2cdb68: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:01.567497 28937 tablet_bootstrap.cc:654] T d208e27cce54405faa75a6aa0a4c6bd4 P b5da4d315109496e97d99f4f5422780f: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:01.568828 28937 log.cc:824] T d208e27cce54405faa75a6aa0a4c6bd4 P b5da4d315109496e97d99f4f5422780f: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:01.570272 28938 tablet_bootstrap.cc:492] T d208e27cce54405faa75a6aa0a4c6bd4 P 9699d6741cbe4c0bb0c2561a6e2cdb68: No bootstrap required, opened a new log
I20260928 16:55:01.570698 28938 ts_tablet_manager.cc:1397] T d208e27cce54405faa75a6aa0a4c6bd4 P 9699d6741cbe4c0bb0c2561a6e2cdb68: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260928 16:55:01.572077 28937 tablet_bootstrap.cc:492] T d208e27cce54405faa75a6aa0a4c6bd4 P b5da4d315109496e97d99f4f5422780f: No bootstrap required, opened a new log
I20260928 16:55:01.572381 28937 ts_tablet_manager.cc:1397] T d208e27cce54405faa75a6aa0a4c6bd4 P b5da4d315109496e97d99f4f5422780f: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260928 16:55:01.572011 28936 raft_consensus.cc:348] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3835a835706486f85fd8baa80ce48da" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 46717 } } peers { permanent_uuid: "b5da4d315109496e97d99f4f5422780f" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 36231 } } peers { permanent_uuid: "9699d6741cbe4c0bb0c2561a6e2cdb68" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34563 } }
I20260928 16:55:01.572546 28936 raft_consensus.cc:374] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:01.572724 28936 raft_consensus.cc:729] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3835a835706486f85fd8baa80ce48da, State: Initialized, Role: FOLLOWER
I20260928 16:55:01.573596 28936 consensus_queue.cc:260] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3835a835706486f85fd8baa80ce48da" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 46717 } } peers { permanent_uuid: "b5da4d315109496e97d99f4f5422780f" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 36231 } } peers { permanent_uuid: "9699d6741cbe4c0bb0c2561a6e2cdb68" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34563 } }
I20260928 16:55:01.575024 28788 heartbeater.cc:503] Master 127.9.93.62:41723 was elected leader, sending a full tablet report...
I20260928 16:55:01.575861 28936 ts_tablet_manager.cc:1428] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da: Time spent starting tablet: real 0.015s user 0.008s sys 0.007s
I20260928 16:55:01.581674 28938 raft_consensus.cc:348] T d208e27cce54405faa75a6aa0a4c6bd4 P 9699d6741cbe4c0bb0c2561a6e2cdb68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3835a835706486f85fd8baa80ce48da" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 46717 } } peers { permanent_uuid: "b5da4d315109496e97d99f4f5422780f" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 36231 } } peers { permanent_uuid: "9699d6741cbe4c0bb0c2561a6e2cdb68" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34563 } }
I20260928 16:55:01.581996 28937 raft_consensus.cc:348] T d208e27cce54405faa75a6aa0a4c6bd4 P b5da4d315109496e97d99f4f5422780f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3835a835706486f85fd8baa80ce48da" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 46717 } } peers { permanent_uuid: "b5da4d315109496e97d99f4f5422780f" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 36231 } } peers { permanent_uuid: "9699d6741cbe4c0bb0c2561a6e2cdb68" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34563 } }
I20260928 16:55:01.582285 28938 raft_consensus.cc:374] T d208e27cce54405faa75a6aa0a4c6bd4 P 9699d6741cbe4c0bb0c2561a6e2cdb68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:01.582391 28938 raft_consensus.cc:729] T d208e27cce54405faa75a6aa0a4c6bd4 P 9699d6741cbe4c0bb0c2561a6e2cdb68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9699d6741cbe4c0bb0c2561a6e2cdb68, State: Initialized, Role: FOLLOWER
I20260928 16:55:01.582423 28937 raft_consensus.cc:374] T d208e27cce54405faa75a6aa0a4c6bd4 P b5da4d315109496e97d99f4f5422780f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:01.582523 28937 raft_consensus.cc:729] T d208e27cce54405faa75a6aa0a4c6bd4 P b5da4d315109496e97d99f4f5422780f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5da4d315109496e97d99f4f5422780f, State: Initialized, Role: FOLLOWER
I20260928 16:55:01.583146 28937 consensus_queue.cc:260] T d208e27cce54405faa75a6aa0a4c6bd4 P b5da4d315109496e97d99f4f5422780f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3835a835706486f85fd8baa80ce48da" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 46717 } } peers { permanent_uuid: "b5da4d315109496e97d99f4f5422780f" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 36231 } } peers { permanent_uuid: "9699d6741cbe4c0bb0c2561a6e2cdb68" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34563 } }
I20260928 16:55:01.583143 28938 consensus_queue.cc:260] T d208e27cce54405faa75a6aa0a4c6bd4 P 9699d6741cbe4c0bb0c2561a6e2cdb68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3835a835706486f85fd8baa80ce48da" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 46717 } } peers { permanent_uuid: "b5da4d315109496e97d99f4f5422780f" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 36231 } } peers { permanent_uuid: "9699d6741cbe4c0bb0c2561a6e2cdb68" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34563 } }
I20260928 16:55:01.584932 28918 heartbeater.cc:503] Master 127.9.93.62:41723 was elected leader, sending a full tablet report...
I20260928 16:55:01.585332 28937 ts_tablet_manager.cc:1428] T d208e27cce54405faa75a6aa0a4c6bd4 P b5da4d315109496e97d99f4f5422780f: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260928 16:55:01.585635 28938 ts_tablet_manager.cc:1428] T d208e27cce54405faa75a6aa0a4c6bd4 P 9699d6741cbe4c0bb0c2561a6e2cdb68: Time spent starting tablet: real 0.015s user 0.008s sys 0.006s
W20260928 16:55:01.695147 28789 tablet.cc:2369] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:55:01.704058 28919 tablet.cc:2369] T d208e27cce54405faa75a6aa0a4c6bd4 P 9699d6741cbe4c0bb0c2561a6e2cdb68: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:55:01.704564 28658 tablet.cc:2369] T d208e27cce54405faa75a6aa0a4c6bd4 P b5da4d315109496e97d99f4f5422780f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:55:01.827900 28942 raft_consensus.cc:482] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:55:01.828212 28942 raft_consensus.cc:504] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3835a835706486f85fd8baa80ce48da" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 46717 } } peers { permanent_uuid: "b5da4d315109496e97d99f4f5422780f" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 36231 } } peers { permanent_uuid: "9699d6741cbe4c0bb0c2561a6e2cdb68" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34563 } }
I20260928 16:55:01.830130 28942 leader_election.cc:288] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5da4d315109496e97d99f4f5422780f (127.9.93.1:36231), 9699d6741cbe4c0bb0c2561a6e2cdb68 (127.9.93.3:34563)
I20260928 16:55:01.838773 28935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:01.831550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:57611 (local address 127.9.93.3:34563)
0928 16:55:01.831769 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:01.831775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:01.832017 (+ 242us) server_negotiation.cc:407] Connection header received
0928 16:55:01.832178 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:01.832182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:01.832309 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:01.832417 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:55:01.832909 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.833856 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.836312 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.837571 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.837694 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:01.837792 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:01.838371 (+ 579us) server_negotiation.cc:299] Negotiation successful
0928 16:55:01.838520 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 16:55:01.838788 28933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:01.830693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:39633 (local address 127.9.93.1:36231)
0928 16:55:01.830913 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:01.830919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:01.831224 (+ 305us) server_negotiation.cc:407] Connection header received
0928 16:55:01.831461 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:01.831468 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:01.831570 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:01.831696 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:55:01.832467 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.833497 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.836232 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.837518 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.837683 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:01.837793 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:01.838315 (+ 522us) server_negotiation.cc:299] Negotiation successful
0928 16:55:01.838491 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 16:55:01.838852 28945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:01.830415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:36231 (local address 127.9.93.2:39633)
0928 16:55:01.831078 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:01.831098 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:01.831262 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:01.831798 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:01.831803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:01.831839 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:55:01.832280 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:01.832292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.833644 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.833648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:01.835939 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:01.835959 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.837776 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.837784 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:01.838065 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:01.838126 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 16:55:01.838226 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 16:55:01.838409 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":104,"spinlock_wait_cycles":37760,"thread_start_us":314,"threads_started":1}
I20260928 16:55:01.838912 28946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:01.831216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:34563 (local address 127.9.93.2:57611)
0928 16:55:01.831869 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:01.831886 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:01.832046 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:01.832444 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:01.832447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:01.832473 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:55:01.832763 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:01.832771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.834024 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.834029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:01.836102 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:01.836116 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.837791 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.837798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:01.838029 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:01.838119 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 16:55:01.838226 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 16:55:01.838409 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":24,"thread_start_us":323,"threads_started":1}
I20260928 16:55:01.840096 28612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d208e27cce54405faa75a6aa0a4c6bd4" candidate_uuid: "f3835a835706486f85fd8baa80ce48da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5da4d315109496e97d99f4f5422780f" is_pre_election: true
I20260928 16:55:01.840368 28873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d208e27cce54405faa75a6aa0a4c6bd4" candidate_uuid: "f3835a835706486f85fd8baa80ce48da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9699d6741cbe4c0bb0c2561a6e2cdb68" is_pre_election: true
I20260928 16:55:01.840683 28612 raft_consensus.cc:2454] T d208e27cce54405faa75a6aa0a4c6bd4 P b5da4d315109496e97d99f4f5422780f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3835a835706486f85fd8baa80ce48da in term 0.
I20260928 16:55:01.840893 28873 raft_consensus.cc:2454] T d208e27cce54405faa75a6aa0a4c6bd4 P 9699d6741cbe4c0bb0c2561a6e2cdb68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3835a835706486f85fd8baa80ce48da in term 0.
I20260928 16:55:01.841444 28677 leader_election.cc:302] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5da4d315109496e97d99f4f5422780f, f3835a835706486f85fd8baa80ce48da; no voters:
I20260928 16:55:01.841862 28942 raft_consensus.cc:2781] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:55:01.842017 28942 raft_consensus.cc:482] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:55:01.842127 28942 raft_consensus.cc:3037] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:01.844635 28942 raft_consensus.cc:504] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3835a835706486f85fd8baa80ce48da" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 46717 } } peers { permanent_uuid: "b5da4d315109496e97d99f4f5422780f" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 36231 } } peers { permanent_uuid: "9699d6741cbe4c0bb0c2561a6e2cdb68" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34563 } }
I20260928 16:55:01.845301 28942 leader_election.cc:288] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da [CANDIDATE]: Term 1 election: Requested vote from peers b5da4d315109496e97d99f4f5422780f (127.9.93.1:36231), 9699d6741cbe4c0bb0c2561a6e2cdb68 (127.9.93.3:34563)
I20260928 16:55:01.845994 28873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d208e27cce54405faa75a6aa0a4c6bd4" candidate_uuid: "f3835a835706486f85fd8baa80ce48da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9699d6741cbe4c0bb0c2561a6e2cdb68"
I20260928 16:55:01.845995 28612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d208e27cce54405faa75a6aa0a4c6bd4" candidate_uuid: "f3835a835706486f85fd8baa80ce48da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5da4d315109496e97d99f4f5422780f"
I20260928 16:55:01.846208 28873 raft_consensus.cc:3037] T d208e27cce54405faa75a6aa0a4c6bd4 P 9699d6741cbe4c0bb0c2561a6e2cdb68 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:01.846216 28612 raft_consensus.cc:3037] T d208e27cce54405faa75a6aa0a4c6bd4 P b5da4d315109496e97d99f4f5422780f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:01.848704 28873 raft_consensus.cc:2454] T d208e27cce54405faa75a6aa0a4c6bd4 P 9699d6741cbe4c0bb0c2561a6e2cdb68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3835a835706486f85fd8baa80ce48da in term 1.
I20260928 16:55:01.848711 28612 raft_consensus.cc:2454] T d208e27cce54405faa75a6aa0a4c6bd4 P b5da4d315109496e97d99f4f5422780f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3835a835706486f85fd8baa80ce48da in term 1.
I20260928 16:55:01.849242 28677 leader_election.cc:302] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5da4d315109496e97d99f4f5422780f, f3835a835706486f85fd8baa80ce48da; no voters:
I20260928 16:55:01.849601 28942 raft_consensus.cc:2781] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:55:01.850061 28942 raft_consensus.cc:686] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da [term 1 LEADER]: Becoming Leader. State: Replica: f3835a835706486f85fd8baa80ce48da, State: Running, Role: LEADER
I20260928 16:55:01.850646 28942 consensus_queue.cc:237] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3835a835706486f85fd8baa80ce48da" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 46717 } } peers { permanent_uuid: "b5da4d315109496e97d99f4f5422780f" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 36231 } } peers { permanent_uuid: "9699d6741cbe4c0bb0c2561a6e2cdb68" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34563 } }
I20260928 16:55:01.855859 28471 catalog_manager.cc:5495] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da reported cstate change: term changed from 0 to 1, leader changed from <none> to f3835a835706486f85fd8baa80ce48da (127.9.93.2). New cstate: current_term: 1 leader_uuid: "f3835a835706486f85fd8baa80ce48da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3835a835706486f85fd8baa80ce48da" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 46717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5da4d315109496e97d99f4f5422780f" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 36231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9699d6741cbe4c0bb0c2561a6e2cdb68" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20260928 16:55:01.898661 28931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:01.892955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50204 (local address 127.9.93.2:46717)
0928 16:55:01.893173 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:01.893179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:01.893201 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:55:01.893377 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:01.893381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:01.893475 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:01.893621 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:55:01.894053 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.894763 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.896244 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.896675 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.896800 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:01.897447 (+ 647us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:55:01.898013 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:55:01.898194 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:01.898355 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 16:55:01.898403 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:55:01.908200 28873 raft_consensus.cc:1261] T d208e27cce54405faa75a6aa0a4c6bd4 P 9699d6741cbe4c0bb0c2561a6e2cdb68 [term 1 FOLLOWER]: Refusing update from remote peer f3835a835706486f85fd8baa80ce48da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:55:01.908200 28612 raft_consensus.cc:1261] T d208e27cce54405faa75a6aa0a4c6bd4 P b5da4d315109496e97d99f4f5422780f [term 1 FOLLOWER]: Refusing update from remote peer f3835a835706486f85fd8baa80ce48da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:55:01.909286 28947 consensus_queue.cc:1035] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da [LEADER]: Connected to new peer: Peer: permanent_uuid: "9699d6741cbe4c0bb0c2561a6e2cdb68" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:55:01.909693 28942 consensus_queue.cc:1035] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5da4d315109496e97d99f4f5422780f" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 36231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:55:01.943367 28470 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:55:01.943743 28470 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:55:01.948698 28470 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d894570c16bf4261aa93c0e9d598ed5e: Sending DeleteTablet for 3 replicas of tablet d208e27cce54405faa75a6aa0a4c6bd4
I20260928 16:55:01.950284 28723 tablet_service.cc:1514] Processing DeleteTablet for tablet d208e27cce54405faa75a6aa0a4c6bd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:55:01 UTC) from {username='slave'} at 127.0.0.1:50196
I20260928 16:55:01.950740 28592 tablet_service.cc:1514] Processing DeleteTablet for tablet d208e27cce54405faa75a6aa0a4c6bd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:55:01 UTC) from {username='slave'} at 127.0.0.1:35782
I20260928 16:55:01.951499 28853 tablet_service.cc:1514] Processing DeleteTablet for tablet d208e27cce54405faa75a6aa0a4c6bd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:55:01 UTC) from {username='slave'} at 127.0.0.1:42132
I20260928 16:55:01.955065 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 28530
I20260928 16:55:01.960515 28962 tablet_replica.cc:330] T d208e27cce54405faa75a6aa0a4c6bd4 P 9699d6741cbe4c0bb0c2561a6e2cdb68: stopping tablet replica
I20260928 16:55:01.961694 28962 raft_consensus.cc:2229] T d208e27cce54405faa75a6aa0a4c6bd4 P 9699d6741cbe4c0bb0c2561a6e2cdb68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:55:01.963111 28962 raft_consensus.cc:2258] T d208e27cce54405faa75a6aa0a4c6bd4 P 9699d6741cbe4c0bb0c2561a6e2cdb68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:55:01.966897 28962 ts_tablet_manager.cc:1905] T d208e27cce54405faa75a6aa0a4c6bd4 P 9699d6741cbe4c0bb0c2561a6e2cdb68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:55:01.969326 28961 tablet_replica.cc:330] T d208e27cce54405faa75a6aa0a4c6bd4 P b5da4d315109496e97d99f4f5422780f: stopping tablet replica
I20260928 16:55:01.969537 28960 tablet_replica.cc:330] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da: stopping tablet replica
I20260928 16:55:01.970032 28960 raft_consensus.cc:2229] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:55:01.970021 28961 raft_consensus.cc:2229] T d208e27cce54405faa75a6aa0a4c6bd4 P b5da4d315109496e97d99f4f5422780f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:55:01.970726 28961 raft_consensus.cc:2258] T d208e27cce54405faa75a6aa0a4c6bd4 P b5da4d315109496e97d99f4f5422780f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:55:01.970860 28960 raft_consensus.cc:2258] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:55:01.973472 28961 ts_tablet_manager.cc:1905] T d208e27cce54405faa75a6aa0a4c6bd4 P b5da4d315109496e97d99f4f5422780f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:55:01.973479 28960 ts_tablet_manager.cc:1905] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:55:01.979153 28962 ts_tablet_manager.cc:1918] T d208e27cce54405faa75a6aa0a4c6bd4 P 9699d6741cbe4c0bb0c2561a6e2cdb68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:55:01.979501 28962 log.cc:1196] T d208e27cce54405faa75a6aa0a4c6bd4 P 9699d6741cbe4c0bb0c2561a6e2cdb68: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/d208e27cce54405faa75a6aa0a4c6bd4
I20260928 16:55:01.980211 28962 ts_tablet_manager.cc:1939] T d208e27cce54405faa75a6aa0a4c6bd4 P 9699d6741cbe4c0bb0c2561a6e2cdb68: Deleting consensus metadata
I20260928 16:55:01.981137 28961 ts_tablet_manager.cc:1918] T d208e27cce54405faa75a6aa0a4c6bd4 P b5da4d315109496e97d99f4f5422780f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:55:01.981426 28961 log.cc:1196] T d208e27cce54405faa75a6aa0a4c6bd4 P b5da4d315109496e97d99f4f5422780f: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-0/wal/wals/d208e27cce54405faa75a6aa0a4c6bd4
I20260928 16:55:01.982092 28961 ts_tablet_manager.cc:1939] T d208e27cce54405faa75a6aa0a4c6bd4 P b5da4d315109496e97d99f4f5422780f: Deleting consensus metadata
I20260928 16:55:01.982585 28456 catalog_manager.cc:4841] TS 9699d6741cbe4c0bb0c2561a6e2cdb68 (127.9.93.3:34563): tablet d208e27cce54405faa75a6aa0a4c6bd4 (table test-table [id=9de55b390c484b2fbd6e1e9c4eb644e4]) successfully deleted
I20260928 16:55:01.984233 28456 catalog_manager.cc:4841] TS b5da4d315109496e97d99f4f5422780f (127.9.93.1:36231): tablet d208e27cce54405faa75a6aa0a4c6bd4 (table test-table [id=9de55b390c484b2fbd6e1e9c4eb644e4]) successfully deleted
I20260928 16:55:01.985342 28933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:01.957528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35796 (local address 127.9.93.1:36231)
0928 16:55:01.957741 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:01.957747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:01.968557 (+ 10810us) server_negotiation.cc:407] Connection header received
0928 16:55:01.968802 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:01.968808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:01.968958 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:01.969104 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:01.969875 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.971146 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.980871 (+ 9725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:01.981589 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:01.981718 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:01.983433 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:01.983446 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:01.983467 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:01.984385 (+ 918us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:01.984458 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:01.984488 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:01.984619 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:01.984807 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 16:55:01.984950 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 16:55:01.986189 28652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:55:01.987144 28960 ts_tablet_manager.cc:1918] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:55:01.987479 28960 log.cc:1196] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/d208e27cce54405faa75a6aa0a4c6bd4
I20260928 16:55:01.988157 28960 ts_tablet_manager.cc:1939] T d208e27cce54405faa75a6aa0a4c6bd4 P f3835a835706486f85fd8baa80ce48da: Deleting consensus metadata
I20260928 16:55:01.990537 28456 catalog_manager.cc:4841] TS f3835a835706486f85fd8baa80ce48da (127.9.93.2:46717): tablet d208e27cce54405faa75a6aa0a4c6bd4 (table test-table [id=9de55b390c484b2fbd6e1e9c4eb644e4]) successfully deleted
I20260928 16:55:02.109063 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 28530
I20260928 16:55:02.125792 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 28661
I20260928 16:55:02.134703 28931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:02.126726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50216 (local address 127.9.93.2:46717)
0928 16:55:02.126902 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:02.126907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:02.126927 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:55:02.127149 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:02.127153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:02.127257 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:02.127448 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:02.127982 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:02.128806 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:02.130931 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:02.131490 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:02.131614 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:02.133010 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:02.133020 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:02.133038 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:02.133793 (+ 755us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:02.133858 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:02.133882 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:02.133979 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:02.134197 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 16:55:02.134306 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:55:02.135491 28783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:55:02.251935 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 28661
I20260928 16:55:02.269421 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 28791
I20260928 16:55:02.280220 28935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:02.270260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42146 (local address 127.9.93.3:34563)
0928 16:55:02.270445 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:02.270450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:02.270472 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:55:02.270689 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:02.270694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:02.270782 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:02.270873 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:02.271333 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:02.272093 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:02.273506 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:02.273919 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:02.274037 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:02.278621 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:02.278632 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:02.278650 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:02.279298 (+ 648us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:02.279399 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:02.279429 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:02.279546 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:02.279697 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 16:55:02.279834 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 16:55:02.281065 28913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:55:02.371132 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 28791
I20260928 16:55:02.385799 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 28440
I20260928 16:55:02.387013 28501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:55:02.500221 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 28440
2026-09-28T16:55:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3248 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:55:02 dist-test-slave-wzcb krb5kdc[28970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:55:02 dist-test-slave-wzcb krb5kdc[28970](info): set up 2 sockets
Sep 28 16:55:02 dist-test-slave-wzcb krb5kdc[28970](info): commencing operation
krb5kdc: starting...
W20260928 16:55:03.545908 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:55:03 dist-test-slave-wzcb krb5kdc[28970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:55:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:55:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:55:03.839994 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:35905
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:35905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:55:04.210840 28983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:04.211215 28983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:04.211383 28983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:04.219462 28983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:04.219595 28983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:04.219671 28983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:04.219759 28983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:04.219823 28983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:04.230834 28983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:35905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:35905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:04.232470 28983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:04.233982 28983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:04.246755 28989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:04.246721 28991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:04.246793 28988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:04.247642 28983 server_base.cc:1034] running on GCE node
I20260928 16:55:04.248564 28983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:04.250234 28983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:04.251420 28983 hybrid_clock.cc:648] HybridClock initialized: now 1790614504251351 us; error 49 us; skew 500 ppm
I20260928 16:55:04.251829 28983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:04 dist-test-slave-wzcb krb5kdc[28970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614504, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:04.255167 28983 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:55:04.256847 28983 webserver.cc:460] Webserver started at http://127.9.93.62:45615/ using document root <none> and password file <none>
I20260928 16:55:04.257635 28983 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:04.257764 28983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:04.258117 28983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:04.261622 28983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "edcc924b886749ec94c1926b96bffdad"
format_stamp: "Formatted at 2026-09-28 16:55:04 on dist-test-slave-wzcb"
server_key: "3f1e4928607a055ff277ac650c299422"
server_key_iv: "009f3f2842ba1559f2dafe268989e0f3"
server_key_version: "encryptionkey@0"
I20260928 16:55:04.262455 28983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "edcc924b886749ec94c1926b96bffdad"
format_stamp: "Formatted at 2026-09-28 16:55:04 on dist-test-slave-wzcb"
server_key: "3f1e4928607a055ff277ac650c299422"
server_key_iv: "009f3f2842ba1559f2dafe268989e0f3"
server_key_version: "encryptionkey@0"
I20260928 16:55:04.268544 28983 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 16:55:04.272651 28998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:04.274310 28983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260928 16:55:04.274519 28983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "edcc924b886749ec94c1926b96bffdad"
format_stamp: "Formatted at 2026-09-28 16:55:04 on dist-test-slave-wzcb"
server_key: "3f1e4928607a055ff277ac650c299422"
server_key_iv: "009f3f2842ba1559f2dafe268989e0f3"
server_key_version: "encryptionkey@0"
I20260928 16:55:04.274745 28983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:04.287232 28983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:04.292302 28983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:04.292621 28983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:04.311640 28983 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:35905
I20260928 16:55:04.311659 29049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:35905 every 8 connection(s)
I20260928 16:55:04.313309 28983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:55:04.317873 29050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:04.320693 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 28983
I20260928 16:55:04.320861 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:55:04.321329 9588 external_mini_cluster.cc:1428] Setting key 153463024a502f75d85d864f2603be08
I20260928 16:55:04.332564 29050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edcc924b886749ec94c1926b96bffdad: Bootstrap starting.
I20260928 16:55:04.336618 29050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edcc924b886749ec94c1926b96bffdad: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:04.337785 29050 log.cc:824] T 00000000000000000000000000000000 P edcc924b886749ec94c1926b96bffdad: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:55:04 dist-test-slave-wzcb krb5kdc[28970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:55:04.341816 29050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edcc924b886749ec94c1926b96bffdad: No bootstrap required, opened a new log
I20260928 16:55:04.349599 29053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:04.323523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55968 (local address 127.9.93.62:35905)
0928 16:55:04.324427 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:04.324446 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:04.324504 (+ 58us) server_negotiation.cc:407] Connection header received
0928 16:55:04.327001 (+ 2497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:04.327038 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:04.327738 (+ 700us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:04.328329 (+ 591us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:04.330144 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:04.332216 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:04.333802 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:04.335008 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:04.335207 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:04.339750 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:04.339762 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:04.339784 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:04.339838 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:04.344036 (+ 4198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:04.344960 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:04.344974 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:04.344984 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:04.345249 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:04.345658 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:04.345664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:04.345667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:04.346524 (+ 857us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:04.346847 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:04.347374 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 16:55:04.347959 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":363,"threads_started":1}
I20260928 16:55:04.352451 29050 raft_consensus.cc:348] T 00000000000000000000000000000000 P edcc924b886749ec94c1926b96bffdad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcc924b886749ec94c1926b96bffdad" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 35905 } }
I20260928 16:55:04.352934 29050 raft_consensus.cc:374] T 00000000000000000000000000000000 P edcc924b886749ec94c1926b96bffdad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:04.353071 29050 raft_consensus.cc:729] T 00000000000000000000000000000000 P edcc924b886749ec94c1926b96bffdad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edcc924b886749ec94c1926b96bffdad, State: Initialized, Role: FOLLOWER
I20260928 16:55:04.353765 29050 consensus_queue.cc:260] T 00000000000000000000000000000000 P edcc924b886749ec94c1926b96bffdad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcc924b886749ec94c1926b96bffdad" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 35905 } }
I20260928 16:55:04.354041 29050 raft_consensus.cc:388] T 00000000000000000000000000000000 P edcc924b886749ec94c1926b96bffdad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:55:04.354182 29050 raft_consensus.cc:482] T 00000000000000000000000000000000 P edcc924b886749ec94c1926b96bffdad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:55:04.354354 29050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edcc924b886749ec94c1926b96bffdad [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:04.357052 29050 raft_consensus.cc:504] T 00000000000000000000000000000000 P edcc924b886749ec94c1926b96bffdad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcc924b886749ec94c1926b96bffdad" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 35905 } }
I20260928 16:55:04.357695 29050 leader_election.cc:302] T 00000000000000000000000000000000 P edcc924b886749ec94c1926b96bffdad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edcc924b886749ec94c1926b96bffdad; no voters:
I20260928 16:55:04.358366 29050 leader_election.cc:288] T 00000000000000000000000000000000 P edcc924b886749ec94c1926b96bffdad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:55:04.358523 29055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edcc924b886749ec94c1926b96bffdad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:55:04.359203 29055 raft_consensus.cc:686] T 00000000000000000000000000000000 P edcc924b886749ec94c1926b96bffdad [term 1 LEADER]: Becoming Leader. State: Replica: edcc924b886749ec94c1926b96bffdad, State: Running, Role: LEADER
I20260928 16:55:04.359844 29055 consensus_queue.cc:237] T 00000000000000000000000000000000 P edcc924b886749ec94c1926b96bffdad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcc924b886749ec94c1926b96bffdad" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 35905 } }
I20260928 16:55:04.360684 29050 sys_catalog.cc:564] T 00000000000000000000000000000000 P edcc924b886749ec94c1926b96bffdad [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:55:04.363888 29056 sys_catalog.cc:455] T 00000000000000000000000000000000 P edcc924b886749ec94c1926b96bffdad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edcc924b886749ec94c1926b96bffdad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcc924b886749ec94c1926b96bffdad" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 35905 } } }
I20260928 16:55:04.364252 29056 sys_catalog.cc:458] T 00000000000000000000000000000000 P edcc924b886749ec94c1926b96bffdad [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:04.365044 29057 sys_catalog.cc:455] T 00000000000000000000000000000000 P edcc924b886749ec94c1926b96bffdad [sys.catalog]: SysCatalogTable state changed. Reason: New leader edcc924b886749ec94c1926b96bffdad. Latest consensus state: current_term: 1 leader_uuid: "edcc924b886749ec94c1926b96bffdad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcc924b886749ec94c1926b96bffdad" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 35905 } } }
I20260928 16:55:04.365303 29057 sys_catalog.cc:458] T 00000000000000000000000000000000 P edcc924b886749ec94c1926b96bffdad [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:04.366638 29063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:55:04.373159 29063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:55:04.383910 29063 catalog_manager.cc:1347] Generated new cluster ID: 1414aa98203b4b6293dd19a8b9865039
I20260928 16:55:04.384070 29063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:55:04.395378 29063 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:55:04.396883 29063 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:55:04.406606 29063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edcc924b886749ec94c1926b96bffdad: Generated new TSK 0
I20260928 16:55:04.408068 29063 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
Entry for principal kudu/127.9.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab.
I20260928 16:55:04.483170 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:35905
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:55:04.856860 29076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:04.857221 29076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:04.857375 29076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:04.865890 29076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:04.866074 29076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:04.866274 29076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:55:04.877785 29076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.1.keytab
--principal=kudu/127.9.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:35905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:04.879617 29076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:04.881240 29076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:04.896359 29082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:04.896318 29084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:04.896384 29081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:04.897188 29076 server_base.cc:1034] running on GCE node
I20260928 16:55:04.897883 29076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:04.898999 29076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:04.900220 29076 hybrid_clock.cc:648] HybridClock initialized: now 1790614504900139 us; error 69 us; skew 500 ppm
I20260928 16:55:04.900596 29076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:04 dist-test-slave-wzcb krb5kdc[28970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614504, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:04.904111 29076 init.cc:377] Logged in from keytab as kudu/127.9.93.1@KRBTEST.COM (short username kudu)
I20260928 16:55:04.905890 29076 webserver.cc:460] Webserver started at http://127.9.93.1:41317/ using document root <none> and password file <none>
I20260928 16:55:04.906733 29076 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:04.906857 29076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:04.907188 29076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:04.910583 29076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "34f6f189c0e34604a9722b8c1d5c29f5"
format_stamp: "Formatted at 2026-09-28 16:55:04 on dist-test-slave-wzcb"
server_key: "4ae77417e20941b888abfe03ad2f09f4"
server_key_iv: "dee7864052f0092fe5dface606d09d30"
server_key_version: "encryptionkey@0"
I20260928 16:55:04.911535 29076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "34f6f189c0e34604a9722b8c1d5c29f5"
format_stamp: "Formatted at 2026-09-28 16:55:04 on dist-test-slave-wzcb"
server_key: "4ae77417e20941b888abfe03ad2f09f4"
server_key_iv: "dee7864052f0092fe5dface606d09d30"
server_key_version: "encryptionkey@0"
I20260928 16:55:04.917713 29076 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:55:04.922075 29091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:04.923918 29076 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:55:04.924134 29076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "34f6f189c0e34604a9722b8c1d5c29f5"
format_stamp: "Formatted at 2026-09-28 16:55:04 on dist-test-slave-wzcb"
server_key: "4ae77417e20941b888abfe03ad2f09f4"
server_key_iv: "dee7864052f0092fe5dface606d09d30"
server_key_version: "encryptionkey@0"
I20260928 16:55:04.924387 29076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:04.961127 29076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:04.965901 29076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:04.966223 29076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:04.967406 29076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:55:04.968987 29076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:55:04.969071 29076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:04.969210 29076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:55:04.969266 29076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:05.003005 29076 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:33181
I20260928 16:55:05.003062 29203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:33181 every 8 connection(s)
I20260928 16:55:05.004552 29076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:55:05.006443 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 29076
I20260928 16:55:05.006666 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:55:05.007340 9588 external_mini_cluster.cc:1428] Setting key 60cd5e3dc8236b92a281d429870523de
Sep 28 16:55:05 dist-test-slave-wzcb krb5kdc[28970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614504, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.1@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:55:05.031828 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:05.008103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:42421 (local address 127.9.93.62:35905)
0928 16:55:05.008666 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:05.008671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:05.010169 (+ 1498us) server_negotiation.cc:407] Connection header received
0928 16:55:05.013120 (+ 2951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:05.013125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:05.013217 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:05.013334 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:05.015884 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:05.016678 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:05.019249 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:05.019759 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:05.019887 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:05.024859 (+ 4972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:05.024865 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:05.024869 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:05.024925 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:05.027705 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:05.028959 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:05.028964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:05.028967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:05.029118 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:05.029681 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:05.029685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:05.029687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:05.030089 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:05.030238 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:05.031266 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0928 16:55:05.031532 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":277,"threads_started":1}
I20260928 16:55:05.033504 29207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:05.008884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:35905 (local address 127.9.93.1:42421)
0928 16:55:05.009971 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:05.010029 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:05.012610 (+ 2581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:05.013695 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:05.013708 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:05.014575 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:05.015582 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:05.015615 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:05.016903 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:05.016908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:05.019016 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:05.019037 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:05.020056 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:05.020063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:05.020412 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:05.021235 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:55:05.021284 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:05.024533 (+ 3249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:05.027914 (+ 3381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:05.027927 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:05.027950 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:05.028779 (+ 829us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:05.029243 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:05.029248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:05.029251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:05.029549 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:05.030243 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:05.030252 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:05.030792 (+ 540us) client_negotiation.cc:769] Sending connection context
0928 16:55:05.031163 (+ 371us) client_negotiation.cc:240] Negotiation successful
0928 16:55:05.031887 (+ 724us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":485,"threads_started":1}
I20260928 16:55:05.035557 29204 heartbeater.cc:347] Connected to a master server at 127.9.93.62:35905
I20260928 16:55:05.036005 29204 heartbeater.cc:464] Registering TS with master...
I20260928 16:55:05.037166 29204 heartbeater.cc:511] Master 127.9.93.62:35905 requested a full tablet report, sending...
I20260928 16:55:05.040282 29015 ts_manager.cc:194] Registered new tserver with Master: 34f6f189c0e34604a9722b8c1d5c29f5 (127.9.93.1:33181)
WARNING: no policy specified for kudu/127.9.93.2@KRBTEST.COM; defaulting to no policy
I20260928 16:55:05.042388 29015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.1@KRBTEST.COM'} at 127.9.93.1:42421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
Entry for principal kudu/127.9.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab.
I20260928 16:55:05.078583 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:35905
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:55:05.443979 29210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:05.444298 29210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:05.444398 29210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:05.451982 29210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:05.452118 29210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:05.452239 29210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:55:05.462744 29210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.2.keytab
--principal=kudu/127.9.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:35905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:05.464270 29210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:05.465761 29210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:05.481242 29216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:05.481176 29215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:05.481153 29218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:05.485133 29210 server_base.cc:1034] running on GCE node
I20260928 16:55:05.485888 29210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:05.487098 29210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:05.488368 29210 hybrid_clock.cc:648] HybridClock initialized: now 1790614505488301 us; error 53 us; skew 500 ppm
I20260928 16:55:05.488796 29210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:05 dist-test-slave-wzcb krb5kdc[28970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614505, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:05.492612 29210 init.cc:377] Logged in from keytab as kudu/127.9.93.2@KRBTEST.COM (short username kudu)
I20260928 16:55:05.494346 29210 webserver.cc:460] Webserver started at http://127.9.93.2:36877/ using document root <none> and password file <none>
I20260928 16:55:05.495180 29210 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:05.495337 29210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:05.495685 29210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:05.499006 29210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "2c3c08d963f948048a7acb2eb8cc1556"
format_stamp: "Formatted at 2026-09-28 16:55:05 on dist-test-slave-wzcb"
server_key: "80676d8c838e0bbbca49a39638ef7b98"
server_key_iv: "42f2788e520d6ee5d19b02d9ad11f75c"
server_key_version: "encryptionkey@0"
I20260928 16:55:05.499953 29210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "2c3c08d963f948048a7acb2eb8cc1556"
format_stamp: "Formatted at 2026-09-28 16:55:05 on dist-test-slave-wzcb"
server_key: "80676d8c838e0bbbca49a39638ef7b98"
server_key_iv: "42f2788e520d6ee5d19b02d9ad11f75c"
server_key_version: "encryptionkey@0"
I20260928 16:55:05.505919 29210 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260928 16:55:05.510156 29225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:05.511993 29210 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 16:55:05.512177 29210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "2c3c08d963f948048a7acb2eb8cc1556"
format_stamp: "Formatted at 2026-09-28 16:55:05 on dist-test-slave-wzcb"
server_key: "80676d8c838e0bbbca49a39638ef7b98"
server_key_iv: "42f2788e520d6ee5d19b02d9ad11f75c"
server_key_version: "encryptionkey@0"
I20260928 16:55:05.512403 29210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:05.543924 29210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:05.549352 29210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:05.549642 29210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:05.550750 29210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:55:05.552335 29210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:55:05.552423 29210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:05.552523 29210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:55:05.552575 29210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:05.583362 29210 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:44129
I20260928 16:55:05.583461 29337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:44129 every 8 connection(s)
I20260928 16:55:05.584808 29210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:55:05.589985 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 29210
I20260928 16:55:05.590152 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:55:05.590672 9588 external_mini_cluster.cc:1428] Setting key aa4d47a6a9a42191e06389bc12c551b2
Sep 28 16:55:05 dist-test-slave-wzcb krb5kdc[28970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614505, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.2@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:55:05.607259 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:05.587640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:52041 (local address 127.9.93.62:35905)
0928 16:55:05.588158 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:05.588165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:05.589295 (+ 1130us) server_negotiation.cc:407] Connection header received
0928 16:55:05.591487 (+ 2192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:05.591493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:05.591594 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:05.591692 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:05.593446 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:05.594229 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:05.596018 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:05.596481 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:05.596605 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:05.600384 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:05.600392 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:05.600396 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:05.600443 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:05.603535 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:05.604592 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:05.604596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:05.604599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:05.604755 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:05.605277 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:05.605281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:05.605283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:05.605605 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:05.605749 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:05.606566 (+ 817us) server_negotiation.cc:299] Negotiation successful
0928 16:55:05.606839 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":240,"threads_started":1}
I20260928 16:55:05.608438 29341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:05.588334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:35905 (local address 127.9.93.2:52041)
0928 16:55:05.589102 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:05.589156 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:05.591110 (+ 1954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:05.591992 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:05.592002 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:05.592588 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:05.593247 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:05.593266 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:05.594407 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:05.594411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:05.595854 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:05.595869 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:05.596731 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:05.596738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:05.597001 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:05.597689 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:55:05.597714 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:05.600069 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:05.603730 (+ 3661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:05.603742 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:05.603766 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:05.604462 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:05.604903 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:05.604907 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:05.604909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:05.605138 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:05.605747 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:05.605756 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:05.606164 (+ 408us) client_negotiation.cc:769] Sending connection context
0928 16:55:05.606434 (+ 270us) client_negotiation.cc:240] Negotiation successful
0928 16:55:05.607076 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":307,"threads_started":1}
I20260928 16:55:05.610185 29338 heartbeater.cc:347] Connected to a master server at 127.9.93.62:35905
I20260928 16:55:05.610605 29338 heartbeater.cc:464] Registering TS with master...
I20260928 16:55:05.611707 29338 heartbeater.cc:511] Master 127.9.93.62:35905 requested a full tablet report, sending...
I20260928 16:55:05.613628 29015 ts_manager.cc:194] Registered new tserver with Master: 2c3c08d963f948048a7acb2eb8cc1556 (127.9.93.2:44129)
I20260928 16:55:05.614548 29015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.2@KRBTEST.COM'} at 127.9.93.2:52041
WARNING: no policy specified for kudu/127.9.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
Entry for principal kudu/127.9.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab.
I20260928 16:55:05.658646 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:35905
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:55:06.032375 29344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:06.032747 29344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:06.032893 29344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:06.040874 29344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:06.041038 29344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:06.041213 29344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:55:06.046300 29204 heartbeater.cc:503] Master 127.9.93.62:35905 was elected leader, sending a full tablet report...
I20260928 16:55:06.053020 29344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.3.keytab
--principal=kudu/127.9.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:35905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:06.054520 29344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:06.056193 29344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:06.071940 29352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:06.071837 29350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:06.071854 29349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:06.072786 29344 server_base.cc:1034] running on GCE node
I20260928 16:55:06.073518 29344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:06.074618 29344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:06.075843 29344 hybrid_clock.cc:648] HybridClock initialized: now 1790614506075770 us; error 62 us; skew 500 ppm
I20260928 16:55:06.076175 29344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:06 dist-test-slave-wzcb krb5kdc[28970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614506, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:06.079909 29344 init.cc:377] Logged in from keytab as kudu/127.9.93.3@KRBTEST.COM (short username kudu)
I20260928 16:55:06.081624 29344 webserver.cc:460] Webserver started at http://127.9.93.3:42573/ using document root <none> and password file <none>
I20260928 16:55:06.082474 29344 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:06.082604 29344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:06.083014 29344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:06.086504 29344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "60c87e9fdde24cbfa4a74e38817aa596"
format_stamp: "Formatted at 2026-09-28 16:55:06 on dist-test-slave-wzcb"
server_key: "7d40973884d1f895a413de4740d03936"
server_key_iv: "c7c6a1c5e740b80717bf40ca7c146b97"
server_key_version: "encryptionkey@0"
I20260928 16:55:06.087602 29344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "60c87e9fdde24cbfa4a74e38817aa596"
format_stamp: "Formatted at 2026-09-28 16:55:06 on dist-test-slave-wzcb"
server_key: "7d40973884d1f895a413de4740d03936"
server_key_iv: "c7c6a1c5e740b80717bf40ca7c146b97"
server_key_version: "encryptionkey@0"
I20260928 16:55:06.094353 29344 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:55:06.098974 29359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:06.100872 29344 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:55:06.101075 29344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "60c87e9fdde24cbfa4a74e38817aa596"
format_stamp: "Formatted at 2026-09-28 16:55:06 on dist-test-slave-wzcb"
server_key: "7d40973884d1f895a413de4740d03936"
server_key_iv: "c7c6a1c5e740b80717bf40ca7c146b97"
server_key_version: "encryptionkey@0"
I20260928 16:55:06.101310 29344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:06.116243 29344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:06.121173 29344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:06.121505 29344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:06.122659 29344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:55:06.124281 29344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:55:06.124370 29344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:06.124504 29344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:55:06.124560 29344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:06.156992 29344 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:36143
I20260928 16:55:06.157034 29471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:36143 every 8 connection(s)
I20260928 16:55:06.158457 29344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:55:06.159631 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 29344
I20260928 16:55:06.159844 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:55:06.160526 9588 external_mini_cluster.cc:1428] Setting key 576abd12aefbd2bf8e39f46d6afa131c
Sep 28 16:55:06 dist-test-slave-wzcb krb5kdc[28970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614506, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.3@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:55:06.183604 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:06.162041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:41525 (local address 127.9.93.62:35905)
0928 16:55:06.162571 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:06.162577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:06.164052 (+ 1475us) server_negotiation.cc:407] Connection header received
0928 16:55:06.166055 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:06.166060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:06.166160 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:06.166257 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:06.168250 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.169079 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.170898 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.171380 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.171508 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:06.176016 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:06.176022 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:06.176026 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:06.176069 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:06.179457 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:06.180459 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:06.180464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:06.180467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:06.180617 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:06.181258 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:06.181262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:06.181265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:06.181609 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:06.181749 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:06.182834 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0928 16:55:06.183208 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":266,"threads_started":1}
I20260928 16:55:06.184708 29475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:06.162835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:35905 (local address 127.9.93.3:41525)
0928 16:55:06.163893 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:06.163952 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:06.165732 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:06.166604 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:06.166619 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:06.167255 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:06.168044 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:06.168066 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.169261 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.169265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:06.170702 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:06.170719 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.171671 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.171681 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:06.171937 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:06.172721 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:55:06.172750 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:06.175665 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:06.179667 (+ 4002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:06.179678 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:06.179696 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:06.180292 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:06.180816 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:06.180822 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:06.180825 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:06.181090 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:06.181763 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:06.181770 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:06.182193 (+ 423us) client_negotiation.cc:769] Sending connection context
0928 16:55:06.182652 (+ 459us) client_negotiation.cc:240] Negotiation successful
0928 16:55:06.183370 (+ 718us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":457,"threads_started":1}
I20260928 16:55:06.186362 29472 heartbeater.cc:347] Connected to a master server at 127.9.93.62:35905
I20260928 16:55:06.186713 29472 heartbeater.cc:464] Registering TS with master...
I20260928 16:55:06.187676 29472 heartbeater.cc:511] Master 127.9.93.62:35905 requested a full tablet report, sending...
I20260928 16:55:06.189529 29015 ts_manager.cc:194] Registered new tserver with Master: 60c87e9fdde24cbfa4a74e38817aa596 (127.9.93.3:36143)
I20260928 16:55:06.190231 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:55:06.190569 29015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.93.3@KRBTEST.COM'} at 127.9.93.3:41525
I20260928 16:55:06.209064 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:06.194883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55982 (local address 127.9.93.62:35905)
0928 16:55:06.195131 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:06.195138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:06.195454 (+ 316us) server_negotiation.cc:407] Connection header received
0928 16:55:06.195631 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:06.195635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:06.195708 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:06.195785 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:06.197419 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.198175 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.199794 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.200292 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.200401 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:06.203018 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:06.203023 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:06.203027 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:06.203068 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:06.205826 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:06.206717 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:06.206721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:06.206724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:06.206866 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:06.207403 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:06.207408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:06.207410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:06.207820 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:06.208065 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:06.208510 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 16:55:06.208742 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 16:55:06.213634 29015 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:55:06.221175 29015 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:55:06.259570 29485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:06.235116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:33181 (local address 127.0.0.1:50242)
0928 16:55:06.236009 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:06.236057 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:06.236290 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:06.250243 (+ 13953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:06.250252 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:06.250292 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:55:06.250801 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:06.250818 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.253214 (+ 2396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.253225 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:06.256018 (+ 2793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:06.256037 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.258382 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.258389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:06.258615 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:06.258698 (+ 83us) client_negotiation.cc:769] Sending connection context
0928 16:55:06.258845 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 16:55:06.259084 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":33,"thread_start_us":430,"threads_started":1}
I20260928 16:55:06.260437 29489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:06.235851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:44129 (local address 127.0.0.1:59774)
0928 16:55:06.249783 (+ 13932us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:06.249809 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:06.249994 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:06.252695 (+ 2701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:06.252701 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:06.252739 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:55:06.253394 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:06.253413 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.255240 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.255249 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:06.257505 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:06.257525 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.259582 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.259588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:06.259763 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:06.259795 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 16:55:06.259877 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 16:55:06.260016 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13759,"mutex_wait_us":60,"thread_start_us":17196,"threads_started":1}
I20260928 16:55:06.260442 29486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:06.235360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50242 (local address 127.9.93.1:33181)
0928 16:55:06.249270 (+ 13910us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:06.249283 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:06.249335 (+ 52us) server_negotiation.cc:407] Connection header received
0928 16:55:06.249492 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:06.249501 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:06.249855 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:06.251077 (+ 1222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:55:06.251384 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.253024 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.256194 (+ 3170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.257793 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.257936 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:06.258225 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:06.259869 (+ 1644us) server_negotiation.cc:299] Negotiation successful
0928 16:55:06.260086 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13689,"thread_start_us":13501,"threads_started":1}
I20260928 16:55:06.260982 29488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:06.236382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59774 (local address 127.9.93.2:44129)
0928 16:55:06.251966 (+ 15584us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:06.251982 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:06.252015 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:55:06.252117 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:06.252125 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:06.252433 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:06.252671 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:55:06.253617 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.255066 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.257705 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.259295 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.259501 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:06.259718 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:06.260466 (+ 748us) server_negotiation.cc:299] Negotiation successful
0928 16:55:06.260667 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15382,"thread_start_us":15270,"threads_started":1}
I20260928 16:55:06.261804 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:06.236606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:36143 (local address 127.0.0.1:39042)
0928 16:55:06.253502 (+ 16896us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:06.253523 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:06.253658 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:06.254730 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:06.254735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:06.254763 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:55:06.255165 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:06.255177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.256888 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.256893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:06.258821 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:06.258838 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.261047 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.261053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:06.261216 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:06.261248 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 16:55:06.261343 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 16:55:06.261491 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16732,"thread_start_us":12825,"threads_started":1}
I20260928 16:55:06.263252 29490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:06.236951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39042 (local address 127.9.93.3:36143)
0928 16:55:06.251867 (+ 14916us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:06.251878 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:06.253794 (+ 1916us) server_negotiation.cc:407] Connection header received
0928 16:55:06.253927 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:06.253937 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:06.254225 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:06.254442 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:55:06.255417 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.256740 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.259034 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.260516 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.260648 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:06.260898 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:06.261439 (+ 541us) server_negotiation.cc:299] Negotiation successful
0928 16:55:06.262901 (+ 1462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14277,"thread_start_us":14073,"threads_started":1}
I20260928 16:55:06.265082 29139 tablet_service.cc:1467] Processing CreateTablet for tablet 02064c7fcb8e4927b9ef7e645c48b04a (DEFAULT_TABLE table=test-table [id=269b31daf7414602b03123bb2d17545a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:55:06.265082 29273 tablet_service.cc:1467] Processing CreateTablet for tablet 02064c7fcb8e4927b9ef7e645c48b04a (DEFAULT_TABLE table=test-table [id=269b31daf7414602b03123bb2d17545a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:55:06.266832 29139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02064c7fcb8e4927b9ef7e645c48b04a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:06.266847 29273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02064c7fcb8e4927b9ef7e645c48b04a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:06.267266 29407 tablet_service.cc:1467] Processing CreateTablet for tablet 02064c7fcb8e4927b9ef7e645c48b04a (DEFAULT_TABLE table=test-table [id=269b31daf7414602b03123bb2d17545a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:55:06.269560 29407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02064c7fcb8e4927b9ef7e645c48b04a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:06.283093 29491 tablet_bootstrap.cc:492] T 02064c7fcb8e4927b9ef7e645c48b04a P 60c87e9fdde24cbfa4a74e38817aa596: Bootstrap starting.
I20260928 16:55:06.284466 29493 tablet_bootstrap.cc:492] T 02064c7fcb8e4927b9ef7e645c48b04a P 34f6f189c0e34604a9722b8c1d5c29f5: Bootstrap starting.
I20260928 16:55:06.287482 29491 tablet_bootstrap.cc:654] T 02064c7fcb8e4927b9ef7e645c48b04a P 60c87e9fdde24cbfa4a74e38817aa596: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:06.288512 29493 tablet_bootstrap.cc:654] T 02064c7fcb8e4927b9ef7e645c48b04a P 34f6f189c0e34604a9722b8c1d5c29f5: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:06.288980 29492 tablet_bootstrap.cc:492] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556: Bootstrap starting.
I20260928 16:55:06.288996 29491 log.cc:824] T 02064c7fcb8e4927b9ef7e645c48b04a P 60c87e9fdde24cbfa4a74e38817aa596: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:06.289956 29493 log.cc:824] T 02064c7fcb8e4927b9ef7e645c48b04a P 34f6f189c0e34604a9722b8c1d5c29f5: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:06.293143 29491 tablet_bootstrap.cc:492] T 02064c7fcb8e4927b9ef7e645c48b04a P 60c87e9fdde24cbfa4a74e38817aa596: No bootstrap required, opened a new log
I20260928 16:55:06.293488 29491 ts_tablet_manager.cc:1397] T 02064c7fcb8e4927b9ef7e645c48b04a P 60c87e9fdde24cbfa4a74e38817aa596: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260928 16:55:06.293871 29492 tablet_bootstrap.cc:654] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:06.294167 29493 tablet_bootstrap.cc:492] T 02064c7fcb8e4927b9ef7e645c48b04a P 34f6f189c0e34604a9722b8c1d5c29f5: No bootstrap required, opened a new log
I20260928 16:55:06.294440 29493 ts_tablet_manager.cc:1397] T 02064c7fcb8e4927b9ef7e645c48b04a P 34f6f189c0e34604a9722b8c1d5c29f5: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 16:55:06.295433 29492 log.cc:824] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:06.299289 29492 tablet_bootstrap.cc:492] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556: No bootstrap required, opened a new log
I20260928 16:55:06.299645 29492 ts_tablet_manager.cc:1397] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260928 16:55:06.302603 29493 raft_consensus.cc:348] T 02064c7fcb8e4927b9ef7e645c48b04a P 34f6f189c0e34604a9722b8c1d5c29f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f6f189c0e34604a9722b8c1d5c29f5" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 33181 } } peers { permanent_uuid: "60c87e9fdde24cbfa4a74e38817aa596" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36143 } } peers { permanent_uuid: "2c3c08d963f948048a7acb2eb8cc1556" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 44129 } }
I20260928 16:55:06.303180 29493 raft_consensus.cc:374] T 02064c7fcb8e4927b9ef7e645c48b04a P 34f6f189c0e34604a9722b8c1d5c29f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:06.303368 29493 raft_consensus.cc:729] T 02064c7fcb8e4927b9ef7e645c48b04a P 34f6f189c0e34604a9722b8c1d5c29f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f6f189c0e34604a9722b8c1d5c29f5, State: Initialized, Role: FOLLOWER
I20260928 16:55:06.304134 29493 consensus_queue.cc:260] T 02064c7fcb8e4927b9ef7e645c48b04a P 34f6f189c0e34604a9722b8c1d5c29f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f6f189c0e34604a9722b8c1d5c29f5" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 33181 } } peers { permanent_uuid: "60c87e9fdde24cbfa4a74e38817aa596" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36143 } } peers { permanent_uuid: "2c3c08d963f948048a7acb2eb8cc1556" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 44129 } }
I20260928 16:55:06.304458 29491 raft_consensus.cc:348] T 02064c7fcb8e4927b9ef7e645c48b04a P 60c87e9fdde24cbfa4a74e38817aa596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f6f189c0e34604a9722b8c1d5c29f5" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 33181 } } peers { permanent_uuid: "60c87e9fdde24cbfa4a74e38817aa596" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36143 } } peers { permanent_uuid: "2c3c08d963f948048a7acb2eb8cc1556" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 44129 } }
I20260928 16:55:06.305014 29491 raft_consensus.cc:374] T 02064c7fcb8e4927b9ef7e645c48b04a P 60c87e9fdde24cbfa4a74e38817aa596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:06.305155 29491 raft_consensus.cc:729] T 02064c7fcb8e4927b9ef7e645c48b04a P 60c87e9fdde24cbfa4a74e38817aa596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60c87e9fdde24cbfa4a74e38817aa596, State: Initialized, Role: FOLLOWER
I20260928 16:55:06.305900 29491 consensus_queue.cc:260] T 02064c7fcb8e4927b9ef7e645c48b04a P 60c87e9fdde24cbfa4a74e38817aa596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f6f189c0e34604a9722b8c1d5c29f5" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 33181 } } peers { permanent_uuid: "60c87e9fdde24cbfa4a74e38817aa596" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36143 } } peers { permanent_uuid: "2c3c08d963f948048a7acb2eb8cc1556" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 44129 } }
I20260928 16:55:06.306111 29493 ts_tablet_manager.cc:1428] T 02064c7fcb8e4927b9ef7e645c48b04a P 34f6f189c0e34604a9722b8c1d5c29f5: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260928 16:55:06.307662 29472 heartbeater.cc:503] Master 127.9.93.62:35905 was elected leader, sending a full tablet report...
I20260928 16:55:06.308389 29491 ts_tablet_manager.cc:1428] T 02064c7fcb8e4927b9ef7e645c48b04a P 60c87e9fdde24cbfa4a74e38817aa596: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260928 16:55:06.311462 29492 raft_consensus.cc:348] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f6f189c0e34604a9722b8c1d5c29f5" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 33181 } } peers { permanent_uuid: "60c87e9fdde24cbfa4a74e38817aa596" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36143 } } peers { permanent_uuid: "2c3c08d963f948048a7acb2eb8cc1556" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 44129 } }
I20260928 16:55:06.312063 29492 raft_consensus.cc:374] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:06.312176 29492 raft_consensus.cc:729] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3c08d963f948048a7acb2eb8cc1556, State: Initialized, Role: FOLLOWER
I20260928 16:55:06.312875 29492 consensus_queue.cc:260] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f6f189c0e34604a9722b8c1d5c29f5" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 33181 } } peers { permanent_uuid: "60c87e9fdde24cbfa4a74e38817aa596" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36143 } } peers { permanent_uuid: "2c3c08d963f948048a7acb2eb8cc1556" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 44129 } }
I20260928 16:55:06.314054 29338 heartbeater.cc:503] Master 127.9.93.62:35905 was elected leader, sending a full tablet report...
I20260928 16:55:06.314803 29492 ts_tablet_manager.cc:1428] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
W20260928 16:55:06.336971 29339 tablet.cc:2369] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:55:06.372455 29499 raft_consensus.cc:482] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:55:06.372825 29499 raft_consensus.cc:504] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f6f189c0e34604a9722b8c1d5c29f5" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 33181 } } peers { permanent_uuid: "60c87e9fdde24cbfa4a74e38817aa596" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36143 } } peers { permanent_uuid: "2c3c08d963f948048a7acb2eb8cc1556" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 44129 } }
I20260928 16:55:06.374676 29499 leader_election.cc:288] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34f6f189c0e34604a9722b8c1d5c29f5 (127.9.93.1:33181), 60c87e9fdde24cbfa4a74e38817aa596 (127.9.93.3:36143)
I20260928 16:55:06.382200 29486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:06.375227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:41543 (local address 127.9.93.1:33181)
0928 16:55:06.375526 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:06.375534 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:06.375917 (+ 383us) server_negotiation.cc:407] Connection header received
0928 16:55:06.376167 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:06.376174 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:06.376268 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:06.376368 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:55:06.377109 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.378156 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.380417 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.381239 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.381371 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:06.381483 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:06.381796 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 16:55:06.381948 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260928 16:55:06.382282 29500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:06.375125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:33181 (local address 127.9.93.2:41543)
0928 16:55:06.375790 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:06.375808 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:06.375989 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:06.376550 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:06.376554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:06.376583 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:55:06.376917 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:06.376928 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.378300 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.378306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:06.380253 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:06.380267 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.381475 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.381481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:06.381631 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:06.381658 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 16:55:06.381742 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 16:55:06.381889 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":381,"threads_started":1}
I20260928 16:55:06.382718 29490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:06.376265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:51185 (local address 127.9.93.3:36143)
0928 16:55:06.376473 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:06.376480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:06.377064 (+ 584us) server_negotiation.cc:407] Connection header received
0928 16:55:06.377334 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:06.377339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:06.377425 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:06.377519 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:55:06.378113 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.379207 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.380980 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.381762 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.381896 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:06.382012 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:06.382307 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 16:55:06.382463 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 16:55:06.382764 29501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:06.375974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:36143 (local address 127.9.93.2:51185)
0928 16:55:06.376967 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:06.376986 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:06.377174 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:06.377550 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:06.377555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:06.377583 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:55:06.377938 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:06.377950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.379381 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.379385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:06.380810 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:06.380823 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.381986 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.381992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:06.382155 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:06.382181 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:55:06.382274 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 16:55:06.382415 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":285,"threads_started":1}
I20260928 16:55:06.383692 29159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02064c7fcb8e4927b9ef7e645c48b04a" candidate_uuid: "2c3c08d963f948048a7acb2eb8cc1556" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34f6f189c0e34604a9722b8c1d5c29f5" is_pre_election: true
I20260928 16:55:06.383701 29427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02064c7fcb8e4927b9ef7e645c48b04a" candidate_uuid: "2c3c08d963f948048a7acb2eb8cc1556" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60c87e9fdde24cbfa4a74e38817aa596" is_pre_election: true
I20260928 16:55:06.384232 29427 raft_consensus.cc:2454] T 02064c7fcb8e4927b9ef7e645c48b04a P 60c87e9fdde24cbfa4a74e38817aa596 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c3c08d963f948048a7acb2eb8cc1556 in term 0.
I20260928 16:55:06.384239 29159 raft_consensus.cc:2454] T 02064c7fcb8e4927b9ef7e645c48b04a P 34f6f189c0e34604a9722b8c1d5c29f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c3c08d963f948048a7acb2eb8cc1556 in term 0.
I20260928 16:55:06.384925 29226 leader_election.cc:302] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c3c08d963f948048a7acb2eb8cc1556, 34f6f189c0e34604a9722b8c1d5c29f5; no voters:
I20260928 16:55:06.385340 29499 raft_consensus.cc:2781] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:55:06.385509 29499 raft_consensus.cc:482] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:55:06.385628 29499 raft_consensus.cc:3037] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:06.388065 29499 raft_consensus.cc:504] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f6f189c0e34604a9722b8c1d5c29f5" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 33181 } } peers { permanent_uuid: "60c87e9fdde24cbfa4a74e38817aa596" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36143 } } peers { permanent_uuid: "2c3c08d963f948048a7acb2eb8cc1556" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 44129 } }
I20260928 16:55:06.388784 29499 leader_election.cc:288] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556 [CANDIDATE]: Term 1 election: Requested vote from peers 34f6f189c0e34604a9722b8c1d5c29f5 (127.9.93.1:33181), 60c87e9fdde24cbfa4a74e38817aa596 (127.9.93.3:36143)
I20260928 16:55:06.389478 29159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02064c7fcb8e4927b9ef7e645c48b04a" candidate_uuid: "2c3c08d963f948048a7acb2eb8cc1556" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34f6f189c0e34604a9722b8c1d5c29f5"
I20260928 16:55:06.389488 29427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02064c7fcb8e4927b9ef7e645c48b04a" candidate_uuid: "2c3c08d963f948048a7acb2eb8cc1556" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60c87e9fdde24cbfa4a74e38817aa596"
I20260928 16:55:06.389711 29159 raft_consensus.cc:3037] T 02064c7fcb8e4927b9ef7e645c48b04a P 34f6f189c0e34604a9722b8c1d5c29f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:06.389765 29427 raft_consensus.cc:3037] T 02064c7fcb8e4927b9ef7e645c48b04a P 60c87e9fdde24cbfa4a74e38817aa596 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:06.392285 29159 raft_consensus.cc:2454] T 02064c7fcb8e4927b9ef7e645c48b04a P 34f6f189c0e34604a9722b8c1d5c29f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c3c08d963f948048a7acb2eb8cc1556 in term 1.
I20260928 16:55:06.392597 29427 raft_consensus.cc:2454] T 02064c7fcb8e4927b9ef7e645c48b04a P 60c87e9fdde24cbfa4a74e38817aa596 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c3c08d963f948048a7acb2eb8cc1556 in term 1.
I20260928 16:55:06.392851 29226 leader_election.cc:302] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c3c08d963f948048a7acb2eb8cc1556, 34f6f189c0e34604a9722b8c1d5c29f5; no voters:
I20260928 16:55:06.393208 29499 raft_consensus.cc:2781] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:55:06.393709 29499 raft_consensus.cc:686] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556 [term 1 LEADER]: Becoming Leader. State: Replica: 2c3c08d963f948048a7acb2eb8cc1556, State: Running, Role: LEADER
I20260928 16:55:06.394235 29499 consensus_queue.cc:237] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f6f189c0e34604a9722b8c1d5c29f5" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 33181 } } peers { permanent_uuid: "60c87e9fdde24cbfa4a74e38817aa596" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36143 } } peers { permanent_uuid: "2c3c08d963f948048a7acb2eb8cc1556" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 44129 } }
I20260928 16:55:06.399261 29015 catalog_manager.cc:5495] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c3c08d963f948048a7acb2eb8cc1556 (127.9.93.2). New cstate: current_term: 1 leader_uuid: "2c3c08d963f948048a7acb2eb8cc1556" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f6f189c0e34604a9722b8c1d5c29f5" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 33181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60c87e9fdde24cbfa4a74e38817aa596" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c3c08d963f948048a7acb2eb8cc1556" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 44129 } health_report { overall_health: HEALTHY } } }
W20260928 16:55:06.410223 29473 tablet.cc:2369] T 02064c7fcb8e4927b9ef7e645c48b04a P 60c87e9fdde24cbfa4a74e38817aa596: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:55:06.420886 29488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:06.414903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59776 (local address 127.9.93.2:44129)
0928 16:55:06.415112 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:06.415118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:06.415142 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:55:06.415448 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:06.415454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:06.415552 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:06.415693 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:55:06.416109 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.416933 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.418471 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.418904 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.419023 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:06.419625 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:55:06.420208 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:55:06.420367 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:06.420483 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 16:55:06.420581 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:55:06.430915 29159 raft_consensus.cc:1261] T 02064c7fcb8e4927b9ef7e645c48b04a P 34f6f189c0e34604a9722b8c1d5c29f5 [term 1 FOLLOWER]: Refusing update from remote peer 2c3c08d963f948048a7acb2eb8cc1556: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:55:06.430915 29427 raft_consensus.cc:1261] T 02064c7fcb8e4927b9ef7e645c48b04a P 60c87e9fdde24cbfa4a74e38817aa596 [term 1 FOLLOWER]: Refusing update from remote peer 2c3c08d963f948048a7acb2eb8cc1556: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:55:06.432042 29502 consensus_queue.cc:1035] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34f6f189c0e34604a9722b8c1d5c29f5" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 33181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:55:06.432508 29499 consensus_queue.cc:1035] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60c87e9fdde24cbfa4a74e38817aa596" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 36143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:55:06.446147 29505 mvcc.cc:205] Tried to move back new op lower bound from 7334357018328764416 to 7334357018195197952. Current Snapshot: MvccSnapshot[applied={T|T < 7334357018328764416}]
I20260928 16:55:06.448740 29507 mvcc.cc:205] Tried to move back new op lower bound from 7334357018328764416 to 7334357018195197952. Current Snapshot: MvccSnapshot[applied={T|T < 7334357018328764416}]
I20260928 16:55:06.466306 29015 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:55:06.466634 29015 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:55:06.470973 29015 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edcc924b886749ec94c1926b96bffdad: Sending DeleteTablet for 3 replicas of tablet 02064c7fcb8e4927b9ef7e645c48b04a
I20260928 16:55:06.472424 29273 tablet_service.cc:1514] Processing DeleteTablet for tablet 02064c7fcb8e4927b9ef7e645c48b04a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:55:06 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:59774
I20260928 16:55:06.472412 29139 tablet_service.cc:1514] Processing DeleteTablet for tablet 02064c7fcb8e4927b9ef7e645c48b04a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:55:06 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:50242
I20260928 16:55:06.472596 29407 tablet_service.cc:1514] Processing DeleteTablet for tablet 02064c7fcb8e4927b9ef7e645c48b04a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:55:06 UTC) from {username='kudu', principal='kudu/127.9.93.62@KRBTEST.COM'} at 127.0.0.1:39042
I20260928 16:55:06.475150 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 29076
I20260928 16:55:06.484305 29515 tablet_replica.cc:330] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556: stopping tablet replica
I20260928 16:55:06.485219 29515 raft_consensus.cc:2229] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:55:06.486294 29515 raft_consensus.cc:2258] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:55:06.487738 29517 tablet_replica.cc:330] T 02064c7fcb8e4927b9ef7e645c48b04a P 60c87e9fdde24cbfa4a74e38817aa596: stopping tablet replica
I20260928 16:55:06.489190 29517 raft_consensus.cc:2229] T 02064c7fcb8e4927b9ef7e645c48b04a P 60c87e9fdde24cbfa4a74e38817aa596 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:55:06.489429 29515 ts_tablet_manager.cc:1905] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:55:06.490127 29516 tablet_replica.cc:330] T 02064c7fcb8e4927b9ef7e645c48b04a P 34f6f189c0e34604a9722b8c1d5c29f5: stopping tablet replica
I20260928 16:55:06.490422 29517 raft_consensus.cc:2258] T 02064c7fcb8e4927b9ef7e645c48b04a P 60c87e9fdde24cbfa4a74e38817aa596 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:55:06.491438 29516 raft_consensus.cc:2229] T 02064c7fcb8e4927b9ef7e645c48b04a P 34f6f189c0e34604a9722b8c1d5c29f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:55:06.492141 29516 raft_consensus.cc:2258] T 02064c7fcb8e4927b9ef7e645c48b04a P 34f6f189c0e34604a9722b8c1d5c29f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:55:06.493870 29517 ts_tablet_manager.cc:1905] T 02064c7fcb8e4927b9ef7e645c48b04a P 60c87e9fdde24cbfa4a74e38817aa596: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:55:06.498394 29515 ts_tablet_manager.cc:1918] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:55:06.498694 29515 log.cc:1196] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/02064c7fcb8e4927b9ef7e645c48b04a
I20260928 16:55:06.498823 29516 ts_tablet_manager.cc:1905] T 02064c7fcb8e4927b9ef7e645c48b04a P 34f6f189c0e34604a9722b8c1d5c29f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:55:06.499521 29515 ts_tablet_manager.cc:1939] T 02064c7fcb8e4927b9ef7e645c48b04a P 2c3c08d963f948048a7acb2eb8cc1556: Deleting consensus metadata
I20260928 16:55:06.502193 29000 catalog_manager.cc:4841] TS 2c3c08d963f948048a7acb2eb8cc1556 (127.9.93.2:44129): tablet 02064c7fcb8e4927b9ef7e645c48b04a (table test-table [id=269b31daf7414602b03123bb2d17545a]) successfully deleted
Sep 28 16:55:06 dist-test-slave-wzcb krb5kdc[28970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.1@KRBTEST.COM
I20260928 16:55:06.506130 29516 ts_tablet_manager.cc:1918] T 02064c7fcb8e4927b9ef7e645c48b04a P 34f6f189c0e34604a9722b8c1d5c29f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:55:06.506497 29516 log.cc:1196] T 02064c7fcb8e4927b9ef7e645c48b04a P 34f6f189c0e34604a9722b8c1d5c29f5: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-0/wal/wals/02064c7fcb8e4927b9ef7e645c48b04a
I20260928 16:55:06.507221 29516 ts_tablet_manager.cc:1939] T 02064c7fcb8e4927b9ef7e645c48b04a P 34f6f189c0e34604a9722b8c1d5c29f5: Deleting consensus metadata
I20260928 16:55:06.509650 28999 catalog_manager.cc:4841] TS 34f6f189c0e34604a9722b8c1d5c29f5 (127.9.93.1:33181): tablet 02064c7fcb8e4927b9ef7e645c48b04a (table test-table [id=269b31daf7414602b03123bb2d17545a]) successfully deleted
I20260928 16:55:06.509852 29517 ts_tablet_manager.cc:1918] T 02064c7fcb8e4927b9ef7e645c48b04a P 60c87e9fdde24cbfa4a74e38817aa596: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:55:06.510038 29517 log.cc:1196] T 02064c7fcb8e4927b9ef7e645c48b04a P 60c87e9fdde24cbfa4a74e38817aa596: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/02064c7fcb8e4927b9ef7e645c48b04a
I20260928 16:55:06.510561 29517 ts_tablet_manager.cc:1939] T 02064c7fcb8e4927b9ef7e645c48b04a P 60c87e9fdde24cbfa4a74e38817aa596: Deleting consensus metadata
I20260928 16:55:06.512595 28999 catalog_manager.cc:4841] TS 60c87e9fdde24cbfa4a74e38817aa596 (127.9.93.3:36143): tablet 02064c7fcb8e4927b9ef7e645c48b04a (table test-table [id=269b31daf7414602b03123bb2d17545a]) successfully deleted
I20260928 16:55:06.514540 29486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:06.480027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50250 (local address 127.9.93.1:33181)
0928 16:55:06.480254 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:06.480261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:06.490683 (+ 10422us) server_negotiation.cc:407] Connection header received
0928 16:55:06.490938 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:06.490945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:06.491076 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:06.491185 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:06.495231 (+ 4046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.496628 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.498791 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.499514 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.499624 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:06.506222 (+ 6598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:06.506245 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:06.506283 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:06.506339 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:06.510761 (+ 4422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:06.511903 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:06.511914 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:06.511918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:06.512120 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:06.512663 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:06.512668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:06.512672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:06.513190 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:06.513527 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:06.513834 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 16:55:06.514102 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 16:55:06.515282 29199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:55:06.613025 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 29076
I20260928 16:55:06.630633 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 29210
Sep 28 16:55:06 dist-test-slave-wzcb krb5kdc[28970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.2@KRBTEST.COM
I20260928 16:55:06.650487 29488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:06.631639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59786 (local address 127.9.93.2:44129)
0928 16:55:06.631819 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:06.631823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:06.631843 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:55:06.631974 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:06.631978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:06.632060 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:06.632150 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:06.634153 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.635049 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.637112 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.637596 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.637720 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:06.643726 (+ 6006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:06.643747 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:06.643769 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:06.643840 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:06.647187 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:06.648086 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:06.648095 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:06.648098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:06.648284 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:06.648725 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:06.648729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:06.648731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:06.649154 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:06.649503 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:06.649858 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 16:55:06.650101 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:55:06.651389 29333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:55:06.767756 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 29210
I20260928 16:55:06.784473 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 29344
Sep 28 16:55:06 dist-test-slave-wzcb krb5kdc[28970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.3@KRBTEST.COM
I20260928 16:55:06.802196 29490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:06.785370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39052 (local address 127.9.93.3:36143)
0928 16:55:06.785574 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:06.785581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:06.785705 (+ 124us) server_negotiation.cc:407] Connection header received
0928 16:55:06.785953 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:06.785960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:06.786054 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:06.786151 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:06.787621 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.788391 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.789780 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:06.790225 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:06.790336 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:06.795090 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:06.795109 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:06.795127 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:06.795182 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:06.798731 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:06.799974 (+ 1243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:06.799987 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:06.799991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:06.800179 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:06.800598 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:06.800600 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:06.800602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:06.801019 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:06.801251 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:06.801643 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 16:55:06.801844 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:55:06.802824 29467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:55:06.893510 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 29344
I20260928 16:55:06.908263 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 28983
I20260928 16:55:06.909461 29045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:55:07.026178 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 28983
2026-09-28T16:55:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4528 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T16:55:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:55:07Z Disabled control of system clock
I20260928 16:55:07.115276 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:39299
--webserver_interface=127.9.93.62
--webserver_port=0
--builtin_ntp_servers=127.9.93.20:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:39299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:55:07.490464 29526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:07.490839 29526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:07.490970 29526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:07.498742 29526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:07.498883 29526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:07.498926 29526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:07.499009 29526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:07.499065 29526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:07.509164 29526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:39299
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:39299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:07.510702 29526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:07.512779 29526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:07.525954 29532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:07.525923 29531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:07.525978 29534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:07.526743 29526 server_base.cc:1034] running on GCE node
I20260928 16:55:07.527760 29526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:07.529487 29526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:07.530741 29526 hybrid_clock.cc:648] HybridClock initialized: now 1790614507530663 us; error 59 us; skew 500 ppm
I20260928 16:55:07.531183 29526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:55:07.540863 29526 webserver.cc:460] Webserver started at http://127.9.93.62:34629/ using document root <none> and password file <none>
I20260928 16:55:07.541767 29526 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:07.541899 29526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:07.542294 29526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:07.545780 29526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "84d4c7fe08c04e6ebdffde3e31286b6f"
format_stamp: "Formatted at 2026-09-28 16:55:07 on dist-test-slave-wzcb"
server_key: "95faa4ffa3f9993d673f58acb05d712e"
server_key_iv: "efbd03c28b8b958c6baf58a3cc2a4111"
server_key_version: "encryptionkey@0"
I20260928 16:55:07.546710 29526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "84d4c7fe08c04e6ebdffde3e31286b6f"
format_stamp: "Formatted at 2026-09-28 16:55:07 on dist-test-slave-wzcb"
server_key: "95faa4ffa3f9993d673f58acb05d712e"
server_key_iv: "efbd03c28b8b958c6baf58a3cc2a4111"
server_key_version: "encryptionkey@0"
I20260928 16:55:07.552783 29526 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 16:55:07.556969 29540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:07.558740 29526 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:55:07.558964 29526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "84d4c7fe08c04e6ebdffde3e31286b6f"
format_stamp: "Formatted at 2026-09-28 16:55:07 on dist-test-slave-wzcb"
server_key: "95faa4ffa3f9993d673f58acb05d712e"
server_key_iv: "efbd03c28b8b958c6baf58a3cc2a4111"
server_key_version: "encryptionkey@0"
I20260928 16:55:07.559217 29526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:07.576296 29526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:07.577315 29526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:07.577651 29526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:07.597465 29526 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:39299
I20260928 16:55:07.597496 29591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:39299 every 8 connection(s)
I20260928 16:55:07.599097 29526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:55:07.603685 29592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:07.606199 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 29526
I20260928 16:55:07.606412 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:55:07.606992 9588 external_mini_cluster.cc:1428] Setting key bfd08ed589d3b3174d1572869a775b04
I20260928 16:55:07.617913 29592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84d4c7fe08c04e6ebdffde3e31286b6f: Bootstrap starting.
I20260928 16:55:07.621817 29592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84d4c7fe08c04e6ebdffde3e31286b6f: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:07.623198 29592 log.cc:824] T 00000000000000000000000000000000 P 84d4c7fe08c04e6ebdffde3e31286b6f: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:07.623484 29595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:07.609363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60752 (local address 127.9.93.62:39299)
0928 16:55:07.610961 (+ 1598us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:07.610976 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:07.611033 (+ 57us) server_negotiation.cc:407] Connection header received
0928 16:55:07.612979 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:07.613010 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:07.613603 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:07.614078 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:07.614823 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:07.616370 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:07.617907 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:07.618790 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:07.619012 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:07.620540 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:07.620549 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:07.620567 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:07.621207 (+ 640us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:07.621316 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:07.621351 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:07.621470 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:07.621788 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 16:55:07.622227 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1033,"thread_start_us":651,"threads_started":1}
I20260928 16:55:07.626739 29592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84d4c7fe08c04e6ebdffde3e31286b6f: No bootstrap required, opened a new log
I20260928 16:55:07.635097 29592 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84d4c7fe08c04e6ebdffde3e31286b6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d4c7fe08c04e6ebdffde3e31286b6f" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39299 } }
I20260928 16:55:07.635653 29592 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84d4c7fe08c04e6ebdffde3e31286b6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:07.635756 29592 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84d4c7fe08c04e6ebdffde3e31286b6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d4c7fe08c04e6ebdffde3e31286b6f, State: Initialized, Role: FOLLOWER
I20260928 16:55:07.636410 29592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84d4c7fe08c04e6ebdffde3e31286b6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d4c7fe08c04e6ebdffde3e31286b6f" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39299 } }
I20260928 16:55:07.636669 29592 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84d4c7fe08c04e6ebdffde3e31286b6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:55:07.636782 29592 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84d4c7fe08c04e6ebdffde3e31286b6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:55:07.636922 29592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84d4c7fe08c04e6ebdffde3e31286b6f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:07.639811 29592 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84d4c7fe08c04e6ebdffde3e31286b6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d4c7fe08c04e6ebdffde3e31286b6f" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39299 } }
I20260928 16:55:07.640388 29592 leader_election.cc:302] T 00000000000000000000000000000000 P 84d4c7fe08c04e6ebdffde3e31286b6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84d4c7fe08c04e6ebdffde3e31286b6f; no voters:
I20260928 16:55:07.641021 29592 leader_election.cc:288] T 00000000000000000000000000000000 P 84d4c7fe08c04e6ebdffde3e31286b6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:55:07.641181 29597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84d4c7fe08c04e6ebdffde3e31286b6f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:55:07.641991 29597 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84d4c7fe08c04e6ebdffde3e31286b6f [term 1 LEADER]: Becoming Leader. State: Replica: 84d4c7fe08c04e6ebdffde3e31286b6f, State: Running, Role: LEADER
I20260928 16:55:07.642571 29597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84d4c7fe08c04e6ebdffde3e31286b6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d4c7fe08c04e6ebdffde3e31286b6f" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39299 } }
I20260928 16:55:07.643376 29592 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84d4c7fe08c04e6ebdffde3e31286b6f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:55:07.645529 29599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84d4c7fe08c04e6ebdffde3e31286b6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84d4c7fe08c04e6ebdffde3e31286b6f. Latest consensus state: current_term: 1 leader_uuid: "84d4c7fe08c04e6ebdffde3e31286b6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d4c7fe08c04e6ebdffde3e31286b6f" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39299 } } }
I20260928 16:55:07.646617 29599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84d4c7fe08c04e6ebdffde3e31286b6f [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:07.645716 29598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84d4c7fe08c04e6ebdffde3e31286b6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84d4c7fe08c04e6ebdffde3e31286b6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d4c7fe08c04e6ebdffde3e31286b6f" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39299 } } }
I20260928 16:55:07.647223 29598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84d4c7fe08c04e6ebdffde3e31286b6f [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:07.652899 29605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:55:07.659041 29605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:55:07.670367 29605 catalog_manager.cc:1347] Generated new cluster ID: 1b857e63179b40fd90cd0123baa4cebd
I20260928 16:55:07.670538 29605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:55:07.694211 29605 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:55:07.695901 29605 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:55:07.704501 29605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84d4c7fe08c04e6ebdffde3e31286b6f: Generated new TSK 0
I20260928 16:55:07.706037 29605 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:55:07.715983 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.1:0
--local_ip_for_outbound_sockets=127.9.93.1
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:39299
--builtin_ntp_servers=127.9.93.20:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:55:08.092785 29616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:08.093137 29616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:08.093254 29616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:08.101557 29616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:08.101696 29616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:08.101850 29616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.1
I20260928 16:55:08.111999 29616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.93.1
--webserver_port=0
--tserver_master_addrs=127.9.93.62:39299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.1
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:08.113526 29616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:08.115878 29616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:08.131273 29622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:08.131335 29624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:08.131510 29621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:08.132306 29616 server_base.cc:1034] running on GCE node
I20260928 16:55:08.133095 29616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:08.134271 29616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:08.135505 29616 hybrid_clock.cc:648] HybridClock initialized: now 1790614508135445 us; error 52 us; skew 500 ppm
I20260928 16:55:08.135890 29616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:55:08.138645 29616 webserver.cc:460] Webserver started at http://127.9.93.1:34035/ using document root <none> and password file <none>
I20260928 16:55:08.139564 29616 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:08.139684 29616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:08.140053 29616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:08.143499 29616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-0/data/instance:
uuid: "8a9fc8d5487742789bbe98bf0a2074e6"
format_stamp: "Formatted at 2026-09-28 16:55:08 on dist-test-slave-wzcb"
server_key: "eda503c7d7c2eb0d3267bca5d49039d4"
server_key_iv: "068115e26cec3c7dde810f2e8f851e58"
server_key_version: "encryptionkey@0"
I20260928 16:55:08.144523 29616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance:
uuid: "8a9fc8d5487742789bbe98bf0a2074e6"
format_stamp: "Formatted at 2026-09-28 16:55:08 on dist-test-slave-wzcb"
server_key: "eda503c7d7c2eb0d3267bca5d49039d4"
server_key_iv: "068115e26cec3c7dde810f2e8f851e58"
server_key_version: "encryptionkey@0"
I20260928 16:55:08.150732 29616 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:55:08.155157 29630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:08.157027 29616 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260928 16:55:08.157246 29616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-0/wal
uuid: "8a9fc8d5487742789bbe98bf0a2074e6"
format_stamp: "Formatted at 2026-09-28 16:55:08 on dist-test-slave-wzcb"
server_key: "eda503c7d7c2eb0d3267bca5d49039d4"
server_key_iv: "068115e26cec3c7dde810f2e8f851e58"
server_key_version: "encryptionkey@0"
I20260928 16:55:08.157508 29616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:08.184360 29616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:08.185402 29616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:08.185719 29616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:08.186892 29616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:55:08.188700 29616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:55:08.188810 29616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:08.188935 29616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:55:08.189019 29616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:08.229473 29616 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.1:38777
I20260928 16:55:08.229557 29742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.1:38777 every 8 connection(s)
I20260928 16:55:08.230978 29616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-0/data/info.pb
I20260928 16:55:08.238210 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 29616
I20260928 16:55:08.238423 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-0/wal/instance
I20260928 16:55:08.239017 9588 external_mini_cluster.cc:1428] Setting key c78f29edfde8c127184d968ffeba13fe
I20260928 16:55:08.242677 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.2:0
--local_ip_for_outbound_sockets=127.9.93.2
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:39299
--builtin_ntp_servers=127.9.93.20:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:55:08.247277 29745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:08.234066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.1:58541 (local address 127.9.93.62:39299)
0928 16:55:08.234591 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:08.234595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:08.235886 (+ 1291us) server_negotiation.cc:407] Connection header received
0928 16:55:08.238029 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:08.238035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:08.238117 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:08.238208 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:08.240066 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:08.241145 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:08.243716 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:08.244226 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:08.244337 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:08.246185 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:08.246191 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:08.246195 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:08.246225 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:08.246286 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:08.246303 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:08.246399 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:08.246718 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 16:55:08.246872 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":265,"threads_started":1}
I20260928 16:55:08.248744 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:08.234626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:39299 (local address 127.9.93.1:58541)
0928 16:55:08.235711 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:08.235754 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:08.237621 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:08.238557 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:08.238571 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:08.238808 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:08.239848 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:08.239874 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:08.241318 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:08.241322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:08.243504 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:08.243525 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:08.244503 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:08.244511 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:08.244850 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:08.245685 (+ 835us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:55:08.245689 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:08.245751 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:55:08.245875 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:08.246429 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:08.246441 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:08.246449 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:55:08.246644 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 16:55:08.247172 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":290,"threads_started":1}
I20260928 16:55:08.250679 29743 heartbeater.cc:347] Connected to a master server at 127.9.93.62:39299
I20260928 16:55:08.251049 29743 heartbeater.cc:464] Registering TS with master...
I20260928 16:55:08.251965 29743 heartbeater.cc:511] Master 127.9.93.62:39299 requested a full tablet report, sending...
I20260928 16:55:08.254621 29557 ts_manager.cc:194] Registered new tserver with Master: 8a9fc8d5487742789bbe98bf0a2074e6 (127.9.93.1:38777)
I20260928 16:55:08.256331 29557 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.93.1:58541
W20260928 16:55:08.609725 29747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:08.610050 29747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:08.610168 29747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:08.618067 29747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:08.618204 29747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:08.618360 29747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.2
I20260928 16:55:08.628276 29747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.93.2
--webserver_port=0
--tserver_master_addrs=127.9.93.62:39299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.2
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:08.629791 29747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:08.632079 29747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:08.647557 29752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:08.647593 29755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:08.647622 29753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:08.648535 29747 server_base.cc:1034] running on GCE node
I20260928 16:55:08.649295 29747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:08.650555 29747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:08.651822 29747 hybrid_clock.cc:648] HybridClock initialized: now 1790614508651739 us; error 71 us; skew 500 ppm
I20260928 16:55:08.652233 29747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:55:08.655095 29747 webserver.cc:460] Webserver started at http://127.9.93.2:33033/ using document root <none> and password file <none>
I20260928 16:55:08.655992 29747 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:08.656121 29747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:08.656492 29747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:08.659986 29747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-1/data/instance:
uuid: "e38d977992654c28852b11255c18c383"
format_stamp: "Formatted at 2026-09-28 16:55:08 on dist-test-slave-wzcb"
server_key: "c3f2e1fec800ceb0cfd60a8ce3ee2b04"
server_key_iv: "87dc54fb5b0afd7d9f361f65a27d9ffb"
server_key_version: "encryptionkey@0"
I20260928 16:55:08.660917 29747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance:
uuid: "e38d977992654c28852b11255c18c383"
format_stamp: "Formatted at 2026-09-28 16:55:08 on dist-test-slave-wzcb"
server_key: "c3f2e1fec800ceb0cfd60a8ce3ee2b04"
server_key_iv: "87dc54fb5b0afd7d9f361f65a27d9ffb"
server_key_version: "encryptionkey@0"
I20260928 16:55:08.667426 29747 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 16:55:08.671885 29761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:08.673661 29747 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:55:08.673885 29747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-1/wal
uuid: "e38d977992654c28852b11255c18c383"
format_stamp: "Formatted at 2026-09-28 16:55:08 on dist-test-slave-wzcb"
server_key: "c3f2e1fec800ceb0cfd60a8ce3ee2b04"
server_key_iv: "87dc54fb5b0afd7d9f361f65a27d9ffb"
server_key_version: "encryptionkey@0"
I20260928 16:55:08.674139 29747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:08.690814 29747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:08.691872 29747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:08.692188 29747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:08.693377 29747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:55:08.695171 29747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:55:08.695278 29747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:08.695425 29747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:55:08.695502 29747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:08.728802 29747 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.2:32997
I20260928 16:55:08.728850 29873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.2:32997 every 8 connection(s)
I20260928 16:55:08.730307 29747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-1/data/info.pb
I20260928 16:55:08.735637 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 29747
I20260928 16:55:08.735852 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-1/wal/instance
I20260928 16:55:08.736449 9588 external_mini_cluster.cc:1428] Setting key e9d8cbd4e22ae49ae5fc20a6c9c4012e
I20260928 16:55:08.739998 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.93.3:0
--local_ip_for_outbound_sockets=127.9.93.3
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:39299
--builtin_ntp_servers=127.9.93.20:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:55:08.745924 29745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:08.733174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.2:48637 (local address 127.9.93.62:39299)
0928 16:55:08.733430 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:08.733437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:08.735113 (+ 1676us) server_negotiation.cc:407] Connection header received
0928 16:55:08.737799 (+ 2686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:08.737805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:08.737902 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:08.738014 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:08.739554 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:08.740408 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:08.742861 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:08.743342 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:08.743456 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:08.745127 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:08.745132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:08.745136 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:08.745159 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:08.745195 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:08.745207 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:08.745276 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:08.745536 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 16:55:08.745641 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":79}
I20260928 16:55:08.747545 29876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:08.733861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:39299 (local address 127.9.93.2:48637)
0928 16:55:08.734907 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:08.734958 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:08.737428 (+ 2470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:08.738265 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:08.738276 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:08.738387 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:08.739349 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:08.739380 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:08.740587 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:08.740592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:08.742680 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:08.742700 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:08.743593 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:08.743600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:08.743900 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:08.744734 (+ 834us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:55:08.744737 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:08.744799 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:55:08.744916 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:08.745295 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:08.745307 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:08.745315 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:55:08.745525 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 16:55:08.745990 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":313,"threads_started":1}
I20260928 16:55:08.749214 29874 heartbeater.cc:347] Connected to a master server at 127.9.93.62:39299
I20260928 16:55:08.749617 29874 heartbeater.cc:464] Registering TS with master...
I20260928 16:55:08.750723 29874 heartbeater.cc:511] Master 127.9.93.62:39299 requested a full tablet report, sending...
I20260928 16:55:08.752637 29557 ts_manager.cc:194] Registered new tserver with Master: e38d977992654c28852b11255c18c383 (127.9.93.2:32997)
I20260928 16:55:08.753697 29557 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.93.2:48637
W20260928 16:55:09.118651 29877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:09.119119 29877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:09.119270 29877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:09.130395 29877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:09.130550 29877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:09.130744 29877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.93.3
I20260928 16:55:09.141040 29877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.93.3
--webserver_port=0
--tserver_master_addrs=127.9.93.62:39299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.93.3
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:09.142925 29877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:09.145191 29877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:09.160326 29882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:09.161093 29885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:09.161093 29883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:09.161393 29877 server_base.cc:1034] running on GCE node
I20260928 16:55:09.162314 29877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:09.163681 29877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:09.165033 29877 hybrid_clock.cc:648] HybridClock initialized: now 1790614509164940 us; error 62 us; skew 500 ppm
I20260928 16:55:09.165504 29877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:55:09.168381 29877 webserver.cc:460] Webserver started at http://127.9.93.3:34275/ using document root <none> and password file <none>
I20260928 16:55:09.169296 29877 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:09.169438 29877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:09.169822 29877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:09.173633 29877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-2/data/instance:
uuid: "f5f0cbc2e52d408e82f16a1ba9ea88b5"
format_stamp: "Formatted at 2026-09-28 16:55:09 on dist-test-slave-wzcb"
server_key: "b98ffab51e255220693c60a1576257c4"
server_key_iv: "909c7a34864b4db16d24a4d694079fe1"
server_key_version: "encryptionkey@0"
I20260928 16:55:09.174685 29877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance:
uuid: "f5f0cbc2e52d408e82f16a1ba9ea88b5"
format_stamp: "Formatted at 2026-09-28 16:55:09 on dist-test-slave-wzcb"
server_key: "b98ffab51e255220693c60a1576257c4"
server_key_iv: "909c7a34864b4db16d24a4d694079fe1"
server_key_version: "encryptionkey@0"
I20260928 16:55:09.181589 29877 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260928 16:55:09.186038 29891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:09.187933 29877 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:55:09.188161 29877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-2/wal
uuid: "f5f0cbc2e52d408e82f16a1ba9ea88b5"
format_stamp: "Formatted at 2026-09-28 16:55:09 on dist-test-slave-wzcb"
server_key: "b98ffab51e255220693c60a1576257c4"
server_key_iv: "909c7a34864b4db16d24a4d694079fe1"
server_key_version: "encryptionkey@0"
I20260928 16:55:09.188431 29877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:09.212990 29877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:09.214144 29877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:09.214490 29877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:09.215796 29877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:55:09.217818 29877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:55:09.217936 29877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:09.218070 29877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:55:09.218146 29877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:09.253883 29877 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.3:44157
I20260928 16:55:09.253980 30003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.3:44157 every 8 connection(s)
I20260928 16:55:09.255532 29877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-2/data/info.pb
I20260928 16:55:09.260159 29743 heartbeater.cc:503] Master 127.9.93.62:39299 was elected leader, sending a full tablet report...
I20260928 16:55:09.264513 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 29877
I20260928 16:55:09.264686 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-2/wal/instance
I20260928 16:55:09.265115 9588 external_mini_cluster.cc:1428] Setting key 93a5d09f340f780a43164a8b7d487dee
I20260928 16:55:09.270366 30006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:09.258457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:41857 (local address 127.9.93.62:39299)
0928 16:55:09.259025 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:09.259030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:09.260380 (+ 1350us) server_negotiation.cc:407] Connection header received
0928 16:55:09.262685 (+ 2305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:09.262689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:09.262761 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:09.262918 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:09.264319 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.265206 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.267271 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.267801 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.267905 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:09.269522 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:09.269528 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:09.269531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:09.269552 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:09.269609 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:09.269623 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:09.269694 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:09.269903 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 16:55:09.269997 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":306,"threads_started":1}
I20260928 16:55:09.271610 30007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:09.258962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:39299 (local address 127.9.93.3:41857)
0928 16:55:09.260130 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:09.260191 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:09.262376 (+ 2185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:09.263229 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:09.263240 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:09.263403 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:09.264094 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:09.264123 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.265395 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.265400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:09.267121 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:09.267136 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.268048 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.268056 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:09.268387 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:09.269131 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:55:09.269133 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:09.269183 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:55:09.269284 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:09.269707 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:09.269715 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:09.269722 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 16:55:09.269884 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 16:55:09.270255 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":324,"threads_started":1}
I20260928 16:55:09.273248 30004 heartbeater.cc:347] Connected to a master server at 127.9.93.62:39299
I20260928 16:55:09.273604 30004 heartbeater.cc:464] Registering TS with master...
I20260928 16:55:09.274500 30004 heartbeater.cc:511] Master 127.9.93.62:39299 requested a full tablet report, sending...
I20260928 16:55:09.276358 29557 ts_manager.cc:194] Registered new tserver with Master: f5f0cbc2e52d408e82f16a1ba9ea88b5 (127.9.93.3:44157)
I20260928 16:55:09.277354 29557 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.93.3:41857
I20260928 16:55:09.281471 9588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:55:09.292416 30006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:09.285995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60754 (local address 127.9.93.62:39299)
0928 16:55:09.286232 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:09.286240 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:09.286636 (+ 396us) server_negotiation.cc:407] Connection header received
0928 16:55:09.286824 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:09.286828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:09.286900 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:09.286991 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:09.287678 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.288412 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.289889 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.290383 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.290498 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:09.291678 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:09.291683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:09.291686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:09.291709 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:09.291765 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:09.291778 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:09.291853 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:09.291952 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 16:55:09.292054 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 16:55:09.297020 29557 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:55:09.304328 29557 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:55:09.347275 30018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:09.318764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.3:44157 (local address 127.0.0.1:58984)
0928 16:55:09.332414 (+ 13650us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:09.332474 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:09.332732 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:09.339408 (+ 6676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:09.339412 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:09.339444 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:55:09.339964 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:09.339980 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.341950 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.341956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:09.344042 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:09.344060 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.346272 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.346277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:09.346445 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:09.346482 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 16:55:09.346601 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 16:55:09.346862 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13315,"mutex_wait_us":81,"thread_start_us":17661,"threads_started":1}
I20260928 16:55:09.347522 30017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:09.318960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:32997 (local address 127.0.0.1:54426)
0928 16:55:09.336792 (+ 17832us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:09.336820 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:09.336967 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:09.339728 (+ 2761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:09.339732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:09.339764 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:55:09.340199 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:09.340211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.342065 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.342070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:09.344611 (+ 2541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:09.344634 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.346686 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.346692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:09.346868 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:09.346899 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 16:55:09.347005 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 16:55:09.347140 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17634,"mutex_wait_us":77,"thread_start_us":12948,"threads_started":1}
I20260928 16:55:09.347868 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:09.319261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54426 (local address 127.9.93.2:32997)
0928 16:55:09.338989 (+ 19728us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:09.338999 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:09.339031 (+ 32us) server_negotiation.cc:407] Connection header received
0928 16:55:09.339156 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:09.339164 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:09.339425 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:09.339635 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:55:09.340407 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.341863 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.344899 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.346470 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.346624 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:09.346877 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:09.347073 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 16:55:09.347282 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19559,"thread_start_us":19365,"threads_started":1}
I20260928 16:55:09.348266 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:09.320403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:38777 (local address 127.0.0.1:33714)
0928 16:55:09.337818 (+ 17415us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:09.337857 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:09.338027 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:09.339062 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:09.339070 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:09.339102 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:55:09.339567 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:09.339582 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.341696 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.341701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:09.344732 (+ 3031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:09.344745 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.347483 (+ 2738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.347489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:09.347660 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:09.347694 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 16:55:09.347778 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 16:55:09.347926 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17258,"mutex_wait_us":152,"thread_start_us":17384,"threads_started":1}
I20260928 16:55:09.348639 30020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:09.319402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58984 (local address 127.9.93.3:44157)
0928 16:55:09.338695 (+ 19293us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:09.338716 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:09.338755 (+ 39us) server_negotiation.cc:407] Connection header received
0928 16:55:09.338868 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:09.338876 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:09.339168 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:09.339424 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:55:09.340230 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.341789 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.344264 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.345676 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.345829 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:09.346158 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:09.348080 (+ 1922us) server_negotiation.cc:299] Negotiation successful
0928 16:55:09.348282 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19086,"thread_start_us":18906,"threads_started":1}
I20260928 16:55:09.350270 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:09.321070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33714 (local address 127.9.93.1:38777)
0928 16:55:09.337327 (+ 16257us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:09.337339 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:09.338119 (+ 780us) server_negotiation.cc:407] Connection header received
0928 16:55:09.338290 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:09.338301 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:09.338637 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:09.338915 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:55:09.339827 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.341543 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.344965 (+ 3422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.347060 (+ 2095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.347205 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:09.348698 (+ 1493us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:09.349700 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0928 16:55:09.349921 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16005,"thread_start_us":16321,"threads_started":1}
I20260928 16:55:09.352913 29809 tablet_service.cc:1467] Processing CreateTablet for tablet 96453664874842a4bb22c17be20fc063 (DEFAULT_TABLE table=test-table [id=d8d49e136fa24b0bbe16e558d01b0967]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:55:09.352923 29939 tablet_service.cc:1467] Processing CreateTablet for tablet 96453664874842a4bb22c17be20fc063 (DEFAULT_TABLE table=test-table [id=d8d49e136fa24b0bbe16e558d01b0967]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:55:09.354615 29809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96453664874842a4bb22c17be20fc063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:09.354722 29939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96453664874842a4bb22c17be20fc063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:09.355705 29678 tablet_service.cc:1467] Processing CreateTablet for tablet 96453664874842a4bb22c17be20fc063 (DEFAULT_TABLE table=test-table [id=d8d49e136fa24b0bbe16e558d01b0967]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:55:09.357442 29678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96453664874842a4bb22c17be20fc063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:09.374663 30025 tablet_bootstrap.cc:492] T 96453664874842a4bb22c17be20fc063 P 8a9fc8d5487742789bbe98bf0a2074e6: Bootstrap starting.
I20260928 16:55:09.377578 30023 tablet_bootstrap.cc:492] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5: Bootstrap starting.
I20260928 16:55:09.380188 30024 tablet_bootstrap.cc:492] T 96453664874842a4bb22c17be20fc063 P e38d977992654c28852b11255c18c383: Bootstrap starting.
I20260928 16:55:09.381609 30025 tablet_bootstrap.cc:654] T 96453664874842a4bb22c17be20fc063 P 8a9fc8d5487742789bbe98bf0a2074e6: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:09.381649 30023 tablet_bootstrap.cc:654] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:09.382870 30025 log.cc:824] T 96453664874842a4bb22c17be20fc063 P 8a9fc8d5487742789bbe98bf0a2074e6: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:09.382941 30023 log.cc:824] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:09.384368 30024 tablet_bootstrap.cc:654] T 96453664874842a4bb22c17be20fc063 P e38d977992654c28852b11255c18c383: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:09.385658 30024 log.cc:824] T 96453664874842a4bb22c17be20fc063 P e38d977992654c28852b11255c18c383: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:09.386695 30025 tablet_bootstrap.cc:492] T 96453664874842a4bb22c17be20fc063 P 8a9fc8d5487742789bbe98bf0a2074e6: No bootstrap required, opened a new log
I20260928 16:55:09.387075 30025 ts_tablet_manager.cc:1397] T 96453664874842a4bb22c17be20fc063 P 8a9fc8d5487742789bbe98bf0a2074e6: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.005s
I20260928 16:55:09.389073 30024 tablet_bootstrap.cc:492] T 96453664874842a4bb22c17be20fc063 P e38d977992654c28852b11255c18c383: No bootstrap required, opened a new log
I20260928 16:55:09.389397 30024 ts_tablet_manager.cc:1397] T 96453664874842a4bb22c17be20fc063 P e38d977992654c28852b11255c18c383: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 16:55:09.394079 30023 tablet_bootstrap.cc:492] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5: No bootstrap required, opened a new log
I20260928 16:55:09.394490 30023 ts_tablet_manager.cc:1397] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20260928 16:55:09.395438 30025 raft_consensus.cc:348] T 96453664874842a4bb22c17be20fc063 P 8a9fc8d5487742789bbe98bf0a2074e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f0cbc2e52d408e82f16a1ba9ea88b5" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44157 } } peers { permanent_uuid: "e38d977992654c28852b11255c18c383" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 32997 } } peers { permanent_uuid: "8a9fc8d5487742789bbe98bf0a2074e6" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38777 } }
I20260928 16:55:09.395911 30025 raft_consensus.cc:374] T 96453664874842a4bb22c17be20fc063 P 8a9fc8d5487742789bbe98bf0a2074e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:09.396018 30025 raft_consensus.cc:729] T 96453664874842a4bb22c17be20fc063 P 8a9fc8d5487742789bbe98bf0a2074e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9fc8d5487742789bbe98bf0a2074e6, State: Initialized, Role: FOLLOWER
I20260928 16:55:09.396703 30025 consensus_queue.cc:260] T 96453664874842a4bb22c17be20fc063 P 8a9fc8d5487742789bbe98bf0a2074e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f0cbc2e52d408e82f16a1ba9ea88b5" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44157 } } peers { permanent_uuid: "e38d977992654c28852b11255c18c383" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 32997 } } peers { permanent_uuid: "8a9fc8d5487742789bbe98bf0a2074e6" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38777 } }
I20260928 16:55:09.399963 30024 raft_consensus.cc:348] T 96453664874842a4bb22c17be20fc063 P e38d977992654c28852b11255c18c383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f0cbc2e52d408e82f16a1ba9ea88b5" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44157 } } peers { permanent_uuid: "e38d977992654c28852b11255c18c383" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 32997 } } peers { permanent_uuid: "8a9fc8d5487742789bbe98bf0a2074e6" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38777 } }
I20260928 16:55:09.400511 30024 raft_consensus.cc:374] T 96453664874842a4bb22c17be20fc063 P e38d977992654c28852b11255c18c383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:09.400609 30024 raft_consensus.cc:729] T 96453664874842a4bb22c17be20fc063 P e38d977992654c28852b11255c18c383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e38d977992654c28852b11255c18c383, State: Initialized, Role: FOLLOWER
I20260928 16:55:09.401032 30025 ts_tablet_manager.cc:1428] T 96453664874842a4bb22c17be20fc063 P 8a9fc8d5487742789bbe98bf0a2074e6: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260928 16:55:09.401439 30024 consensus_queue.cc:260] T 96453664874842a4bb22c17be20fc063 P e38d977992654c28852b11255c18c383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f0cbc2e52d408e82f16a1ba9ea88b5" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44157 } } peers { permanent_uuid: "e38d977992654c28852b11255c18c383" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 32997 } } peers { permanent_uuid: "8a9fc8d5487742789bbe98bf0a2074e6" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38777 } }
I20260928 16:55:09.402758 29874 heartbeater.cc:503] Master 127.9.93.62:39299 was elected leader, sending a full tablet report...
I20260928 16:55:09.403606 30024 ts_tablet_manager.cc:1428] T 96453664874842a4bb22c17be20fc063 P e38d977992654c28852b11255c18c383: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260928 16:55:09.406397 30023 raft_consensus.cc:348] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f0cbc2e52d408e82f16a1ba9ea88b5" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44157 } } peers { permanent_uuid: "e38d977992654c28852b11255c18c383" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 32997 } } peers { permanent_uuid: "8a9fc8d5487742789bbe98bf0a2074e6" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38777 } }
I20260928 16:55:09.406816 30023 raft_consensus.cc:374] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:09.406921 30023 raft_consensus.cc:729] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5f0cbc2e52d408e82f16a1ba9ea88b5, State: Initialized, Role: FOLLOWER
I20260928 16:55:09.407737 30023 consensus_queue.cc:260] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f0cbc2e52d408e82f16a1ba9ea88b5" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44157 } } peers { permanent_uuid: "e38d977992654c28852b11255c18c383" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 32997 } } peers { permanent_uuid: "8a9fc8d5487742789bbe98bf0a2074e6" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38777 } }
I20260928 16:55:09.408895 30004 heartbeater.cc:503] Master 127.9.93.62:39299 was elected leader, sending a full tablet report...
I20260928 16:55:09.409809 30023 ts_tablet_manager.cc:1428] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5: Time spent starting tablet: real 0.015s user 0.017s sys 0.000s
W20260928 16:55:09.482571 29875 tablet.cc:2369] T 96453664874842a4bb22c17be20fc063 P e38d977992654c28852b11255c18c383: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:55:09.483652 29744 tablet.cc:2369] T 96453664874842a4bb22c17be20fc063 P 8a9fc8d5487742789bbe98bf0a2074e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:55:09.496136 30031 raft_consensus.cc:482] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:55:09.496485 30031 raft_consensus.cc:504] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f0cbc2e52d408e82f16a1ba9ea88b5" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44157 } } peers { permanent_uuid: "e38d977992654c28852b11255c18c383" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 32997 } } peers { permanent_uuid: "8a9fc8d5487742789bbe98bf0a2074e6" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38777 } }
I20260928 16:55:09.498385 30031 leader_election.cc:288] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e38d977992654c28852b11255c18c383 (127.9.93.2:32997), 8a9fc8d5487742789bbe98bf0a2074e6 (127.9.93.1:38777)
I20260928 16:55:09.505074 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:09.498917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:49953 (local address 127.9.93.2:32997)
0928 16:55:09.499125 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:09.499130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:09.499156 (+ 26us) server_negotiation.cc:407] Connection header received
0928 16:55:09.499282 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:09.499288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:09.499404 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:09.499503 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:55:09.500003 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.500773 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.502927 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.504110 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.504209 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:09.504284 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:09.504748 (+ 464us) server_negotiation.cc:299] Negotiation successful
0928 16:55:09.504856 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 16:55:09.505294 30007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:09.498656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.2:32997 (local address 127.9.93.3:49953)
0928 16:55:09.498956 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:09.498978 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:09.499124 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:09.499595 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:09.499600 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:09.499638 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:55:09.499828 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:09.499837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.500913 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.500919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:09.502734 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:09.502748 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.504330 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.504339 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:09.504562 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:09.504606 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 16:55:09.504733 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 16:55:09.504903 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":2}
I20260928 16:55:09.506181 30032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:09.498828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:38777 (local address 127.9.93.3:58601)
0928 16:55:09.499804 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:09.499820 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:09.499950 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:09.500261 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:09.500265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:09.500289 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:55:09.500590 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:09.500599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.501951 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.501957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:09.503744 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:09.503761 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.505470 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.505477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:09.505674 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:09.505709 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:55:09.505812 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 16:55:09.505957 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"thread_start_us":592,"threads_started":1}
I20260928 16:55:09.506269 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:09.499173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.93.3:58601 (local address 127.9.93.1:38777)
0928 16:55:09.499447 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:09.499453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:09.499914 (+ 461us) server_negotiation.cc:407] Connection header received
0928 16:55:09.500074 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:09.500078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:09.500149 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:09.500225 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:55:09.500756 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.501782 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.504003 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.505276 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.505432 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:09.505554 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:09.505881 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 16:55:09.506028 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 16:55:09.506657 29829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96453664874842a4bb22c17be20fc063" candidate_uuid: "f5f0cbc2e52d408e82f16a1ba9ea88b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38d977992654c28852b11255c18c383" is_pre_election: true
I20260928 16:55:09.507382 29829 raft_consensus.cc:2454] T 96453664874842a4bb22c17be20fc063 P e38d977992654c28852b11255c18c383 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5f0cbc2e52d408e82f16a1ba9ea88b5 in term 0.
W20260928 16:55:09.507457 30005 tablet.cc:2369] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:55:09.507507 29698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96453664874842a4bb22c17be20fc063" candidate_uuid: "f5f0cbc2e52d408e82f16a1ba9ea88b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9fc8d5487742789bbe98bf0a2074e6" is_pre_election: true
I20260928 16:55:09.508096 29698 raft_consensus.cc:2454] T 96453664874842a4bb22c17be20fc063 P 8a9fc8d5487742789bbe98bf0a2074e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5f0cbc2e52d408e82f16a1ba9ea88b5 in term 0.
I20260928 16:55:09.508162 29893 leader_election.cc:302] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e38d977992654c28852b11255c18c383, f5f0cbc2e52d408e82f16a1ba9ea88b5; no voters:
I20260928 16:55:09.508571 30031 raft_consensus.cc:2781] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:55:09.508714 30031 raft_consensus.cc:482] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:55:09.508831 30031 raft_consensus.cc:3037] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:09.511137 30031 raft_consensus.cc:504] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f0cbc2e52d408e82f16a1ba9ea88b5" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44157 } } peers { permanent_uuid: "e38d977992654c28852b11255c18c383" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 32997 } } peers { permanent_uuid: "8a9fc8d5487742789bbe98bf0a2074e6" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38777 } }
I20260928 16:55:09.511833 30031 leader_election.cc:288] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5 [CANDIDATE]: Term 1 election: Requested vote from peers e38d977992654c28852b11255c18c383 (127.9.93.2:32997), 8a9fc8d5487742789bbe98bf0a2074e6 (127.9.93.1:38777)
I20260928 16:55:09.512344 29829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96453664874842a4bb22c17be20fc063" candidate_uuid: "f5f0cbc2e52d408e82f16a1ba9ea88b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38d977992654c28852b11255c18c383"
I20260928 16:55:09.512423 29698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96453664874842a4bb22c17be20fc063" candidate_uuid: "f5f0cbc2e52d408e82f16a1ba9ea88b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9fc8d5487742789bbe98bf0a2074e6"
I20260928 16:55:09.512548 29829 raft_consensus.cc:3037] T 96453664874842a4bb22c17be20fc063 P e38d977992654c28852b11255c18c383 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:09.512639 29698 raft_consensus.cc:3037] T 96453664874842a4bb22c17be20fc063 P 8a9fc8d5487742789bbe98bf0a2074e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:09.514921 29829 raft_consensus.cc:2454] T 96453664874842a4bb22c17be20fc063 P e38d977992654c28852b11255c18c383 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5f0cbc2e52d408e82f16a1ba9ea88b5 in term 1.
I20260928 16:55:09.514978 29698 raft_consensus.cc:2454] T 96453664874842a4bb22c17be20fc063 P 8a9fc8d5487742789bbe98bf0a2074e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5f0cbc2e52d408e82f16a1ba9ea88b5 in term 1.
I20260928 16:55:09.515417 29893 leader_election.cc:302] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e38d977992654c28852b11255c18c383, f5f0cbc2e52d408e82f16a1ba9ea88b5; no voters:
I20260928 16:55:09.515739 30031 raft_consensus.cc:2781] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:55:09.516234 30031 raft_consensus.cc:686] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5 [term 1 LEADER]: Becoming Leader. State: Replica: f5f0cbc2e52d408e82f16a1ba9ea88b5, State: Running, Role: LEADER
I20260928 16:55:09.516814 30031 consensus_queue.cc:237] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f0cbc2e52d408e82f16a1ba9ea88b5" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44157 } } peers { permanent_uuid: "e38d977992654c28852b11255c18c383" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 32997 } } peers { permanent_uuid: "8a9fc8d5487742789bbe98bf0a2074e6" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38777 } }
I20260928 16:55:09.521912 29557 catalog_manager.cc:5495] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5f0cbc2e52d408e82f16a1ba9ea88b5 (127.9.93.3). New cstate: current_term: 1 leader_uuid: "f5f0cbc2e52d408e82f16a1ba9ea88b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f0cbc2e52d408e82f16a1ba9ea88b5" member_type: VOTER last_known_addr { host: "127.9.93.3" port: 44157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e38d977992654c28852b11255c18c383" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 32997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a9fc8d5487742789bbe98bf0a2074e6" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38777 } health_report { overall_health: UNKNOWN } } }
I20260928 16:55:09.553382 30020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:09.547783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58992 (local address 127.9.93.3:44157)
0928 16:55:09.547997 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:09.548003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:09.548024 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:55:09.548125 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:09.548128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:09.548222 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:09.548366 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:55:09.548793 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.549534 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.550941 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.551464 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.551585 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:09.552138 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:55:09.552772 (+ 634us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:55:09.552931 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:09.553032 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 16:55:09.553125 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 16:55:09.563021 29698 raft_consensus.cc:1261] T 96453664874842a4bb22c17be20fc063 P 8a9fc8d5487742789bbe98bf0a2074e6 [term 1 FOLLOWER]: Refusing update from remote peer f5f0cbc2e52d408e82f16a1ba9ea88b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:55:09.563102 29829 raft_consensus.cc:1261] T 96453664874842a4bb22c17be20fc063 P e38d977992654c28852b11255c18c383 [term 1 FOLLOWER]: Refusing update from remote peer f5f0cbc2e52d408e82f16a1ba9ea88b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:55:09.564188 30033 consensus_queue.cc:1035] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9fc8d5487742789bbe98bf0a2074e6" member_type: VOTER last_known_addr { host: "127.9.93.1" port: 38777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:55:09.564581 30031 consensus_queue.cc:1035] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38d977992654c28852b11255c18c383" member_type: VOTER last_known_addr { host: "127.9.93.2" port: 32997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:55:09.575822 30036 mvcc.cc:205] Tried to move back new op lower bound from 7334357031158353920 to 7334357030984957952. Current Snapshot: MvccSnapshot[applied={T|T < 7334357031158353920}]
I20260928 16:55:09.585391 30038 mvcc.cc:205] Tried to move back new op lower bound from 7334357031158353920 to 7334357030984957952. Current Snapshot: MvccSnapshot[applied={T|T < 7334357031158353920 or (T in {7334357031158353920})}]
I20260928 16:55:09.586050 30037 mvcc.cc:205] Tried to move back new op lower bound from 7334357031158353920 to 7334357030984957952. Current Snapshot: MvccSnapshot[applied={T|T < 7334357031158353920}]
I20260928 16:55:09.593825 29556 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:55:09.594185 29556 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:55:09.598891 29556 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84d4c7fe08c04e6ebdffde3e31286b6f: Sending DeleteTablet for 3 replicas of tablet 96453664874842a4bb22c17be20fc063
I20260928 16:55:09.600641 29939 tablet_service.cc:1514] Processing DeleteTablet for tablet 96453664874842a4bb22c17be20fc063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:55:09 UTC) from {username='slave'} at 127.0.0.1:58984
I20260928 16:55:09.601454 29809 tablet_service.cc:1514] Processing DeleteTablet for tablet 96453664874842a4bb22c17be20fc063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:55:09 UTC) from {username='slave'} at 127.0.0.1:54426
I20260928 16:55:09.601529 29678 tablet_service.cc:1514] Processing DeleteTablet for tablet 96453664874842a4bb22c17be20fc063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:55:09 UTC) from {username='slave'} at 127.0.0.1:33714
I20260928 16:55:09.601887 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 29616
I20260928 16:55:09.610355 30045 tablet_replica.cc:330] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5: stopping tablet replica
I20260928 16:55:09.610918 30045 raft_consensus.cc:2229] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:55:09.611750 30045 raft_consensus.cc:2258] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:55:09.612949 30047 tablet_replica.cc:330] T 96453664874842a4bb22c17be20fc063 P 8a9fc8d5487742789bbe98bf0a2074e6: stopping tablet replica
I20260928 16:55:09.614427 30045 ts_tablet_manager.cc:1905] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:55:09.615164 30047 raft_consensus.cc:2229] T 96453664874842a4bb22c17be20fc063 P 8a9fc8d5487742789bbe98bf0a2074e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:55:09.619827 30047 raft_consensus.cc:2258] T 96453664874842a4bb22c17be20fc063 P 8a9fc8d5487742789bbe98bf0a2074e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:55:09.622257 30046 tablet_replica.cc:330] T 96453664874842a4bb22c17be20fc063 P e38d977992654c28852b11255c18c383: stopping tablet replica
I20260928 16:55:09.622424 30045 ts_tablet_manager.cc:1918] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:55:09.622630 30045 log.cc:1196] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-2/wal/wals/96453664874842a4bb22c17be20fc063
I20260928 16:55:09.623258 30045 ts_tablet_manager.cc:1939] T 96453664874842a4bb22c17be20fc063 P f5f0cbc2e52d408e82f16a1ba9ea88b5: Deleting consensus metadata
I20260928 16:55:09.624208 30046 raft_consensus.cc:2229] T 96453664874842a4bb22c17be20fc063 P e38d977992654c28852b11255c18c383 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:55:09.624531 30047 ts_tablet_manager.cc:1905] T 96453664874842a4bb22c17be20fc063 P 8a9fc8d5487742789bbe98bf0a2074e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:55:09.624797 30046 raft_consensus.cc:2258] T 96453664874842a4bb22c17be20fc063 P e38d977992654c28852b11255c18c383 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:55:09.625715 29541 catalog_manager.cc:4841] TS f5f0cbc2e52d408e82f16a1ba9ea88b5 (127.9.93.3:44157): tablet 96453664874842a4bb22c17be20fc063 (table test-table [id=d8d49e136fa24b0bbe16e558d01b0967]) successfully deleted
I20260928 16:55:09.627767 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:09.603732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33716 (local address 127.9.93.1:38777)
0928 16:55:09.603951 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:09.603958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:09.613358 (+ 9400us) server_negotiation.cc:407] Connection header received
0928 16:55:09.613756 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:09.613763 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:09.613918 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:09.614007 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:09.615783 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.617087 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.620488 (+ 3401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.621241 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.621370 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:09.625688 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:09.625704 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:09.625722 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:09.626594 (+ 872us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:09.626671 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:09.626733 (+ 62us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:09.626866 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:09.627172 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 16:55:09.627359 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 16:55:09.627854 30046 ts_tablet_manager.cc:1905] T 96453664874842a4bb22c17be20fc063 P e38d977992654c28852b11255c18c383: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:55:09.629050 29738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:55:09.635850 30046 ts_tablet_manager.cc:1918] T 96453664874842a4bb22c17be20fc063 P e38d977992654c28852b11255c18c383: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:55:09.636049 30046 log.cc:1196] T 96453664874842a4bb22c17be20fc063 P e38d977992654c28852b11255c18c383: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-1/wal/wals/96453664874842a4bb22c17be20fc063
I20260928 16:55:09.636538 30046 ts_tablet_manager.cc:1939] T 96453664874842a4bb22c17be20fc063 P e38d977992654c28852b11255c18c383: Deleting consensus metadata
I20260928 16:55:09.638322 29542 catalog_manager.cc:4841] TS e38d977992654c28852b11255c18c383 (127.9.93.2:32997): tablet 96453664874842a4bb22c17be20fc063 (table test-table [id=d8d49e136fa24b0bbe16e558d01b0967]) successfully deleted
I20260928 16:55:09.746093 30047 ts_tablet_manager.cc:1918] T 96453664874842a4bb22c17be20fc063 P 8a9fc8d5487742789bbe98bf0a2074e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:55:09.746310 30047 log.cc:1196] T 96453664874842a4bb22c17be20fc063 P 8a9fc8d5487742789bbe98bf0a2074e6: Deleting WAL directory at /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790614166338986-9588-0/minicluster-data/ts-0/wal/wals/96453664874842a4bb22c17be20fc063
I20260928 16:55:09.747231 30047 ts_tablet_manager.cc:1939] T 96453664874842a4bb22c17be20fc063 P 8a9fc8d5487742789bbe98bf0a2074e6: Deleting consensus metadata
I20260928 16:55:09.747237 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 29616
W20260928 16:55:09.764945 29543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.93.1:38777 (error 108)
I20260928 16:55:09.765221 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 29747
I20260928 16:55:09.767436 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:09.766764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:38777 (local address 127.0.0.1:33724)
0928 16:55:09.767034 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:09.767182 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260928 16:55:09.767802 29543 catalog_manager.cc:4584] TS 8a9fc8d5487742789bbe98bf0a2074e6 (127.9.93.1:38777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 96453664874842a4bb22c17be20fc063: Network error: Client connection negotiation failed: client connection to 127.9.93.1:38777: connect: Connection refused (error 111)
I20260928 16:55:09.774444 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:09.766325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54442 (local address 127.9.93.2:32997)
0928 16:55:09.766528 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:09.766533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:09.766626 (+ 93us) server_negotiation.cc:407] Connection header received
0928 16:55:09.766878 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:09.766884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:09.767028 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:09.767147 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:09.767688 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.768566 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.770669 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.771113 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.771218 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:09.772737 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:09.772748 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:09.772766 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:09.773439 (+ 673us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:09.773497 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:09.773520 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:09.773635 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:09.773888 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 16:55:09.774010 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 16:55:09.775346 29869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:55:09.821777 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:09.821141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:38777 (local address 127.0.0.1:33738)
0928 16:55:09.821426 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:09.821553 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 16:55:09.823184 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:09.822687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:38777 (local address 127.0.0.1:33742)
0928 16:55:09.822867 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:09.823002 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 16:55:09.870153 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 29747
I20260928 16:55:09.884213 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:09.883688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:38777 (local address 127.0.0.1:33748)
0928 16:55:09.883921 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:09.884039 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 16:55:09.885398 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:09.885013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:38777 (local address 127.0.0.1:33758)
0928 16:55:09.885173 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:09.885252 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 16:55:09.893024 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 29877
I20260928 16:55:09.901945 30020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:09.894032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59000 (local address 127.9.93.3:44157)
0928 16:55:09.894223 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:09.894228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:09.894343 (+ 115us) server_negotiation.cc:407] Connection header received
0928 16:55:09.894550 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:09.894554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:09.894672 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:09.894760 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:09.895349 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.896327 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.898301 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:09.898793 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:09.898931 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:09.900258 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:09.900268 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:09.900285 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:09.901017 (+ 732us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:09.901076 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:09.901109 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:09.901217 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:09.901445 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 16:55:09.901552 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 16:55:09.902827 29999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:55:09.966162 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:09.965574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:38777 (local address 127.0.0.1:33774)
0928 16:55:09.965838 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:09.965960 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260928 16:55:09.967458 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:09.966946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:38777 (local address 127.0.0.1:33782)
0928 16:55:09.967189 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:09.967271 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260928 16:55:10.026755 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 29877
I20260928 16:55:10.044574 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 29526
I20260928 16:55:10.045866 29587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:55:10.161942 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:10.161134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.1:38777 (local address 127.0.0.1:33790)
0928 16:55:10.161497 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:10.161645 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.1:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211}
I20260928 16:55:10.161959 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 29526
2026-09-28T16:55:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3138 ms)
[----------] 6 tests from EncryptionPolicyTest (23418 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:55:10 dist-test-slave-wzcb krb5kdc[30056](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:55:10 dist-test-slave-wzcb krb5kdc[30056](info): set up 2 sockets
Sep 28 16:55:10 dist-test-slave-wzcb krb5kdc[30056](info): commencing operation
krb5kdc: starting...
W20260928 16:55:11.226378 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:55:11 dist-test-slave-wzcb krb5kdc[30056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:55:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:55:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:55:11.531522 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44249
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:55:11.909240 30069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:11.909590 30069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:11.909757 30069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:11.917559 30069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:11.917713 30069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:11.917801 30069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:11.917891 30069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:11.917946 30069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:11.928890 30069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44249
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:11.930538 30069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:11.932178 30069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:11.945294 30074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:11.945324 30075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:11.945371 30077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:11.946477 30069 server_base.cc:1034] running on GCE node
I20260928 16:55:11.947481 30069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:11.949168 30069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:11.950428 30069 hybrid_clock.cc:648] HybridClock initialized: now 1790614511950381 us; error 33 us; skew 500 ppm
I20260928 16:55:11.950901 30069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:11 dist-test-slave-wzcb krb5kdc[30056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:11.954380 30069 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:55:11.956306 30069 webserver.cc:460] Webserver started at http://127.0.0.1:46321/ using document root <none> and password file <none>
I20260928 16:55:11.957176 30069 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:11.957307 30069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:11.957657 30069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:11.961313 30069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "40f29740ce1b42478b56557248c9cb67"
format_stamp: "Formatted at 2026-09-28 16:55:11 on dist-test-slave-wzcb"
server_key: "38f1e51bffd14389ab56b512587a8557"
server_key_iv: "2c0cb7b2a3030fc34b2c2889d58c8a5e"
server_key_version: "encryptionkey@0"
I20260928 16:55:11.962266 30069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "40f29740ce1b42478b56557248c9cb67"
format_stamp: "Formatted at 2026-09-28 16:55:11 on dist-test-slave-wzcb"
server_key: "38f1e51bffd14389ab56b512587a8557"
server_key_iv: "2c0cb7b2a3030fc34b2c2889d58c8a5e"
server_key_version: "encryptionkey@0"
I20260928 16:55:11.969309 30069 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 16:55:11.973696 30084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:11.975394 30069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:55:11.975605 30069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "40f29740ce1b42478b56557248c9cb67"
format_stamp: "Formatted at 2026-09-28 16:55:11 on dist-test-slave-wzcb"
server_key: "38f1e51bffd14389ab56b512587a8557"
server_key_iv: "2c0cb7b2a3030fc34b2c2889d58c8a5e"
server_key_version: "encryptionkey@0"
I20260928 16:55:11.975826 30069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:11.996978 30069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:12.002058 30069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:12.002382 30069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:12.022729 30069 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44249
I20260928 16:55:12.022750 30135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44249 every 8 connection(s)
I20260928 16:55:12.024482 30069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:55:12.029138 30136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:12.032711 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 30069
I20260928 16:55:12.032971 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:55:12.033891 9588 external_mini_cluster.cc:1428] Setting key 12dbcf31d5fb69a3817c9f387250af7d
I20260928 16:55:12.044198 30136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40f29740ce1b42478b56557248c9cb67: Bootstrap starting.
I20260928 16:55:12.048880 30136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40f29740ce1b42478b56557248c9cb67: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:12.050593 30136 log.cc:824] T 00000000000000000000000000000000 P 40f29740ce1b42478b56557248c9cb67: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:55:12 dist-test-slave-wzcb krb5kdc[30056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 16:55:12.054438 30136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40f29740ce1b42478b56557248c9cb67: No bootstrap required, opened a new log
I20260928 16:55:12.063267 30139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:12.036438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38379 (local address 127.0.0.1:44249)
0928 16:55:12.037351 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:12.037366 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:12.037420 (+ 54us) server_negotiation.cc:407] Connection header received
0928 16:55:12.039392 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:12.039426 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:12.040021 (+ 595us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:12.040462 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:12.042536 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:12.044121 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:12.046114 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:12.046987 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:12.047147 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:12.052903 (+ 5756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:12.052914 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:12.052934 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:12.052979 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:12.057047 (+ 4068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:12.058227 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:12.058244 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:12.058256 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:12.058561 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:12.059165 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:12.059170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:12.059174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:12.060104 (+ 930us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:12.060526 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:12.061049 (+ 523us) server_negotiation.cc:299] Negotiation successful
0928 16:55:12.061535 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":405,"threads_started":1}
I20260928 16:55:12.065384 30136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40f29740ce1b42478b56557248c9cb67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40f29740ce1b42478b56557248c9cb67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44249 } }
I20260928 16:55:12.065821 30136 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40f29740ce1b42478b56557248c9cb67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:12.065960 30136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40f29740ce1b42478b56557248c9cb67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40f29740ce1b42478b56557248c9cb67, State: Initialized, Role: FOLLOWER
I20260928 16:55:12.066666 30136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40f29740ce1b42478b56557248c9cb67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40f29740ce1b42478b56557248c9cb67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44249 } }
I20260928 16:55:12.067055 30136 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40f29740ce1b42478b56557248c9cb67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:55:12.067214 30136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40f29740ce1b42478b56557248c9cb67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:55:12.067428 30136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40f29740ce1b42478b56557248c9cb67 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:12.068516 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:55:12.070914 30136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40f29740ce1b42478b56557248c9cb67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40f29740ce1b42478b56557248c9cb67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44249 } }
I20260928 16:55:12.071779 30136 leader_election.cc:302] T 00000000000000000000000000000000 P 40f29740ce1b42478b56557248c9cb67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40f29740ce1b42478b56557248c9cb67; no voters:
I20260928 16:55:12.072620 30136 leader_election.cc:288] T 00000000000000000000000000000000 P 40f29740ce1b42478b56557248c9cb67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:55:12.072897 30142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40f29740ce1b42478b56557248c9cb67 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:55:12.074570 30142 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40f29740ce1b42478b56557248c9cb67 [term 1 LEADER]: Becoming Leader. State: Replica: 40f29740ce1b42478b56557248c9cb67, State: Running, Role: LEADER
I20260928 16:55:12.075249 30142 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40f29740ce1b42478b56557248c9cb67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40f29740ce1b42478b56557248c9cb67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44249 } }
I20260928 16:55:12.075702 30136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40f29740ce1b42478b56557248c9cb67 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:55:12.079208 30145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40f29740ce1b42478b56557248c9cb67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40f29740ce1b42478b56557248c9cb67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40f29740ce1b42478b56557248c9cb67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44249 } } }
I20260928 16:55:12.079757 30145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40f29740ce1b42478b56557248c9cb67 [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:12.080323 30147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40f29740ce1b42478b56557248c9cb67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40f29740ce1b42478b56557248c9cb67. Latest consensus state: current_term: 1 leader_uuid: "40f29740ce1b42478b56557248c9cb67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40f29740ce1b42478b56557248c9cb67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44249 } } }
I20260928 16:55:12.080632 30147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40f29740ce1b42478b56557248c9cb67 [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:12.087169 30152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:55:12.097265 30152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:55:12.098416 30139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:12.076311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50561 (local address 127.0.0.1:44249)
0928 16:55:12.076507 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:12.076514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:12.076851 (+ 337us) server_negotiation.cc:407] Connection header received
0928 16:55:12.077064 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:12.077070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:12.077155 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:12.077278 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:12.079571 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:12.080948 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:12.083632 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:12.084425 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:12.084555 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:12.089182 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:12.089191 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:12.089197 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:12.089262 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:12.093533 (+ 4271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:12.094874 (+ 1341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:12.094880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:12.094883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:12.095077 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:12.095827 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:12.095833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:12.095836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:12.096312 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:12.096618 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:12.097660 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0928 16:55:12.097956 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:55:12.111600 30152 catalog_manager.cc:1347] Generated new cluster ID: aca5bb49691a40e4afad770f0c6f6170
I20260928 16:55:12.111753 30152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:55:12.147068 30152 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:55:12.148648 30152 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:55:12.166926 30152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40f29740ce1b42478b56557248c9cb67: Generated new TSK 0
I20260928 16:55:12.168017 30152 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:55:12.184649 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 30069
I20260928 16:55:12.186607 30131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:55:12.285913 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 30069
W20260928 16:55:12.304304 30054 connection.cc:535] client connection to 127.0.0.1:44249 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44249): Connection reset by peer (error 104)
2026-09-28T16:55:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2114 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:55:12 dist-test-slave-wzcb krb5kdc[30169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:55:12 dist-test-slave-wzcb krb5kdc[30169](info): set up 2 sockets
Sep 28 16:55:12 dist-test-slave-wzcb krb5kdc[30169](info): commencing operation
krb5kdc: starting...
W20260928 16:55:13.350587 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.004s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:55:13 dist-test-slave-wzcb krb5kdc[30169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:55:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:55:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:55:13.655844 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44335
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:55:14.040131 30182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:14.040519 30182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:14.040684 30182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:14.048952 30182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:14.049126 30182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:14.049212 30182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:14.049309 30182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:14.049386 30182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:14.060774 30182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44335
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:14.062433 30182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:14.064106 30182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:14.078253 30187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:14.078348 30188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:14.078311 30190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:14.079746 30182 server_base.cc:1034] running on GCE node
I20260928 16:55:14.080850 30182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:14.082762 30182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:14.084029 30182 hybrid_clock.cc:648] HybridClock initialized: now 1790614514083963 us; error 61 us; skew 500 ppm
I20260928 16:55:14.084517 30182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:14 dist-test-slave-wzcb krb5kdc[30169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:14.094427 30182 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:55:14.096575 30182 webserver.cc:460] Webserver started at http://127.0.0.1:39321/ using document root <none> and password file <none>
I20260928 16:55:14.097486 30182 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:14.097636 30182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:14.098011 30182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:14.101786 30182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "43c14c51297b46a4aede6fc782fd277f"
format_stamp: "Formatted at 2026-09-28 16:55:14 on dist-test-slave-wzcb"
server_key: "3970572c0d93559dba9d4bb0002ed158"
server_key_iv: "b9c882fdbbf135141b2e9c6d99b37f91"
server_key_version: "encryptionkey@0"
I20260928 16:55:14.102783 30182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "43c14c51297b46a4aede6fc782fd277f"
format_stamp: "Formatted at 2026-09-28 16:55:14 on dist-test-slave-wzcb"
server_key: "3970572c0d93559dba9d4bb0002ed158"
server_key_iv: "b9c882fdbbf135141b2e9c6d99b37f91"
server_key_version: "encryptionkey@0"
I20260928 16:55:14.109639 30182 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 16:55:14.113960 30197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:14.115756 30182 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 16:55:14.115978 30182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "43c14c51297b46a4aede6fc782fd277f"
format_stamp: "Formatted at 2026-09-28 16:55:14 on dist-test-slave-wzcb"
server_key: "3970572c0d93559dba9d4bb0002ed158"
server_key_iv: "b9c882fdbbf135141b2e9c6d99b37f91"
server_key_version: "encryptionkey@0"
I20260928 16:55:14.116211 30182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:14.137712 30182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:14.142838 30182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:14.143180 30182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:14.163708 30182 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44335
I20260928 16:55:14.163697 30248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44335 every 8 connection(s)
I20260928 16:55:14.165328 30182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:55:14.168428 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 30182
I20260928 16:55:14.168650 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:55:14.169260 9588 external_mini_cluster.cc:1428] Setting key 135a7d0627b97fb790b7619a2a04fb72
I20260928 16:55:14.170694 30249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:14.186197 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:14.171555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52017 (local address 127.0.0.1:44335)
0928 16:55:14.172541 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:14.172555 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:14.172605 (+ 50us) server_negotiation.cc:407] Connection header received
0928 16:55:14.174380 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:14.174407 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:14.174975 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:14.175443 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:14.176154 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:14.177653 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:14.179848 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:14.181020 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:14.181258 (+ 238us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:14.182660 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:14.182673 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:14.182703 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:14.183630 (+ 927us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:14.183756 (+ 126us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:14.183800 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:14.184008 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:14.184352 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 16:55:14.184583 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":455,"threads_started":1}
I20260928 16:55:14.187489 30249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c14c51297b46a4aede6fc782fd277f: Bootstrap starting.
I20260928 16:55:14.191151 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:55:14.191917 30249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43c14c51297b46a4aede6fc782fd277f: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:14.193246 30249 log.cc:824] T 00000000000000000000000000000000 P 43c14c51297b46a4aede6fc782fd277f: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:14.197314 30249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c14c51297b46a4aede6fc782fd277f: No bootstrap required, opened a new log
I20260928 16:55:14.204026 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:14.196227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36693 (local address 127.0.0.1:44335)
0928 16:55:14.196446 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:14.196453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:14.196968 (+ 515us) server_negotiation.cc:407] Connection header received
0928 16:55:14.197112 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:14.197118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:14.197205 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:14.197343 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:14.197931 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:14.199079 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:14.200628 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:14.201373 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:14.201532 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:14.202978 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:14.202985 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:14.202990 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:14.203023 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:14.203099 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:14.203144 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:14.203270 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:14.203574 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 16:55:14.203642 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 16:55:14.207293 30249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43c14c51297b46a4aede6fc782fd277f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c14c51297b46a4aede6fc782fd277f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44335 } }
I20260928 16:55:14.207846 30249 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43c14c51297b46a4aede6fc782fd277f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:14.207960 30249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43c14c51297b46a4aede6fc782fd277f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c14c51297b46a4aede6fc782fd277f, State: Initialized, Role: FOLLOWER
I20260928 16:55:14.208678 30249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43c14c51297b46a4aede6fc782fd277f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c14c51297b46a4aede6fc782fd277f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44335 } }
I20260928 16:55:14.208935 30249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43c14c51297b46a4aede6fc782fd277f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:55:14.209088 30249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43c14c51297b46a4aede6fc782fd277f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:55:14.209286 30249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43c14c51297b46a4aede6fc782fd277f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:14.211935 30249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43c14c51297b46a4aede6fc782fd277f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c14c51297b46a4aede6fc782fd277f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44335 } }
I20260928 16:55:14.212599 30249 leader_election.cc:302] T 00000000000000000000000000000000 P 43c14c51297b46a4aede6fc782fd277f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43c14c51297b46a4aede6fc782fd277f; no voters:
I20260928 16:55:14.213359 30249 leader_election.cc:288] T 00000000000000000000000000000000 P 43c14c51297b46a4aede6fc782fd277f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:55:14.213446 30259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43c14c51297b46a4aede6fc782fd277f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:55:14.214213 30259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43c14c51297b46a4aede6fc782fd277f [term 1 LEADER]: Becoming Leader. State: Replica: 43c14c51297b46a4aede6fc782fd277f, State: Running, Role: LEADER
I20260928 16:55:14.214851 30259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43c14c51297b46a4aede6fc782fd277f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c14c51297b46a4aede6fc782fd277f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44335 } }
I20260928 16:55:14.215727 30249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43c14c51297b46a4aede6fc782fd277f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:55:14.218011 30261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c14c51297b46a4aede6fc782fd277f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43c14c51297b46a4aede6fc782fd277f. Latest consensus state: current_term: 1 leader_uuid: "43c14c51297b46a4aede6fc782fd277f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c14c51297b46a4aede6fc782fd277f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44335 } } }
I20260928 16:55:14.218544 30261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c14c51297b46a4aede6fc782fd277f [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:14.218039 30260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c14c51297b46a4aede6fc782fd277f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43c14c51297b46a4aede6fc782fd277f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c14c51297b46a4aede6fc782fd277f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44335 } } }
I20260928 16:55:14.221707 30260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c14c51297b46a4aede6fc782fd277f [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:14.222504 30267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:55:14.227829 30267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:55:14.238660 30267 catalog_manager.cc:1347] Generated new cluster ID: c8038fba963e4a7e96e0de7840885674
I20260928 16:55:14.238801 30267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:55:14.265453 30267 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:55:14.267620 30267 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:55:14.275476 30267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43c14c51297b46a4aede6fc782fd277f: Generated new TSK 0
I20260928 16:55:14.276749 30267 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:55:14.287950 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 30182
W20260928 16:55:14.290036 30244 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52017
W20260928 16:55:14.290932 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 30182: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:55:14.291152 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 30182
W20260928 16:55:14.308544 30167 connection.cc:535] client connection to 127.0.0.1:44335 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44335): Connection reset by peer (error 104)
2026-09-28T16:55:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2003 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:55:14 dist-test-slave-wzcb krb5kdc[30280](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:55:14 dist-test-slave-wzcb krb5kdc[30280](info): set up 2 sockets
Sep 28 16:55:14 dist-test-slave-wzcb krb5kdc[30280](info): commencing operation
krb5kdc: starting...
W20260928 16:55:15.338054 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.004s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:55:15 dist-test-slave-wzcb krb5kdc[30280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:55:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:55:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:55:15.643522 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46639
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:55:16.026160 30293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:16.026547 30293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:16.026701 30293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:16.034597 30293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:16.034761 30293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:16.034854 30293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:16.034947 30293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:16.035012 30293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:16.045898 30293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46639
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:16.047596 30293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:16.049226 30293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:16.063256 30298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:16.063232 30301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:16.063194 30299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:16.064175 30293 server_base.cc:1034] running on GCE node
I20260928 16:55:16.065164 30293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:16.067059 30293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:16.068312 30293 hybrid_clock.cc:648] HybridClock initialized: now 1790614516068246 us; error 64 us; skew 500 ppm
I20260928 16:55:16.068760 30293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:16 dist-test-slave-wzcb krb5kdc[30280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:16.072672 30293 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:55:16.074859 30293 webserver.cc:460] Webserver started at http://127.0.0.1:36423/ using document root <none> and password file <none>
I20260928 16:55:16.075793 30293 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:16.075932 30293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:16.076284 30293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:16.080004 30293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "e9bccfde25d9478ab1a0796c45112c2e"
format_stamp: "Formatted at 2026-09-28 16:55:16 on dist-test-slave-wzcb"
server_key: "34301ed7c59baa91e875237d4abb17ff"
server_key_iv: "4433e60133502b2d44cd4b9a177c98f5"
server_key_version: "encryptionkey@0"
I20260928 16:55:16.080997 30293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "e9bccfde25d9478ab1a0796c45112c2e"
format_stamp: "Formatted at 2026-09-28 16:55:16 on dist-test-slave-wzcb"
server_key: "34301ed7c59baa91e875237d4abb17ff"
server_key_iv: "4433e60133502b2d44cd4b9a177c98f5"
server_key_version: "encryptionkey@0"
I20260928 16:55:16.087769 30293 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260928 16:55:16.092581 30308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:16.094604 30293 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260928 16:55:16.094851 30293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "e9bccfde25d9478ab1a0796c45112c2e"
format_stamp: "Formatted at 2026-09-28 16:55:16 on dist-test-slave-wzcb"
server_key: "34301ed7c59baa91e875237d4abb17ff"
server_key_iv: "4433e60133502b2d44cd4b9a177c98f5"
server_key_version: "encryptionkey@0"
I20260928 16:55:16.095101 30293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:16.102034 30293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:16.107292 30293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:16.107662 30293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:16.128005 30293 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46639
I20260928 16:55:16.128064 30359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46639 every 8 connection(s)
I20260928 16:55:16.129700 30293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:55:16.134683 30360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:16.135799 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 30293
I20260928 16:55:16.135987 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:55:16.136586 9588 external_mini_cluster.cc:1428] Setting key 1e1a34fdefb180bbc25f095760913dd5
I20260928 16:55:16.145987 30363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:16.138772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33001 (local address 127.0.0.1:46639)
0928 16:55:16.139762 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:16.139780 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:16.139835 (+ 55us) server_negotiation.cc:407] Connection header received
0928 16:55:16.141944 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:16.141982 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:16.142472 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:16.142910 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:16.144152 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:16.144159 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:16.144176 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:16.144193 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:16.144261 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:16.144288 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:16.144527 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:16.144643 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 16:55:16.144860 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":455,"threads_started":1}
I20260928 16:55:16.150853 30360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9bccfde25d9478ab1a0796c45112c2e: Bootstrap starting.
I20260928 16:55:16.151087 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:55:16.155633 30360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9bccfde25d9478ab1a0796c45112c2e: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:16.157292 30360 log.cc:824] T 00000000000000000000000000000000 P e9bccfde25d9478ab1a0796c45112c2e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:16.159749 30363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:16.157008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40611 (local address 127.0.0.1:46639)
0928 16:55:16.157232 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:16.157239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:16.157589 (+ 350us) server_negotiation.cc:407] Connection header received
0928 16:55:16.157831 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:16.157835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:16.157898 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:16.158019 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:16.159033 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:16.159038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:16.159041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:16.159057 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:16.159101 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:16.159118 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:16.159214 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:16.159403 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 16:55:16.159462 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 16:55:16.161232 30360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9bccfde25d9478ab1a0796c45112c2e: No bootstrap required, opened a new log
I20260928 16:55:16.169580 30360 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9bccfde25d9478ab1a0796c45112c2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bccfde25d9478ab1a0796c45112c2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46639 } }
I20260928 16:55:16.170065 30360 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9bccfde25d9478ab1a0796c45112c2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:16.170176 30360 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9bccfde25d9478ab1a0796c45112c2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9bccfde25d9478ab1a0796c45112c2e, State: Initialized, Role: FOLLOWER
I20260928 16:55:16.170953 30360 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9bccfde25d9478ab1a0796c45112c2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bccfde25d9478ab1a0796c45112c2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46639 } }
I20260928 16:55:16.171244 30360 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9bccfde25d9478ab1a0796c45112c2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:55:16.171442 30360 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9bccfde25d9478ab1a0796c45112c2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:55:16.171627 30360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9bccfde25d9478ab1a0796c45112c2e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:16.174089 30360 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9bccfde25d9478ab1a0796c45112c2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bccfde25d9478ab1a0796c45112c2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46639 } }
I20260928 16:55:16.174724 30360 leader_election.cc:302] T 00000000000000000000000000000000 P e9bccfde25d9478ab1a0796c45112c2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9bccfde25d9478ab1a0796c45112c2e; no voters:
I20260928 16:55:16.175428 30360 leader_election.cc:288] T 00000000000000000000000000000000 P e9bccfde25d9478ab1a0796c45112c2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:55:16.175591 30370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9bccfde25d9478ab1a0796c45112c2e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:55:16.176354 30370 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9bccfde25d9478ab1a0796c45112c2e [term 1 LEADER]: Becoming Leader. State: Replica: e9bccfde25d9478ab1a0796c45112c2e, State: Running, Role: LEADER
I20260928 16:55:16.177017 30370 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9bccfde25d9478ab1a0796c45112c2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bccfde25d9478ab1a0796c45112c2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46639 } }
I20260928 16:55:16.177835 30360 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9bccfde25d9478ab1a0796c45112c2e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:55:16.180869 30371 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9bccfde25d9478ab1a0796c45112c2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9bccfde25d9478ab1a0796c45112c2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bccfde25d9478ab1a0796c45112c2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46639 } } }
I20260928 16:55:16.180518 30372 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9bccfde25d9478ab1a0796c45112c2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9bccfde25d9478ab1a0796c45112c2e. Latest consensus state: current_term: 1 leader_uuid: "e9bccfde25d9478ab1a0796c45112c2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bccfde25d9478ab1a0796c45112c2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46639 } } }
I20260928 16:55:16.181430 30372 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9bccfde25d9478ab1a0796c45112c2e [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:16.181432 30371 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9bccfde25d9478ab1a0796c45112c2e [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:16.183934 30379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:55:16.189415 30379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:55:16.199723 30379 catalog_manager.cc:1347] Generated new cluster ID: ee6c79840011400ba788d40a1685990d
I20260928 16:55:16.199872 30379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:55:16.219993 30379 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:55:16.220702 30379 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:55:16.235760 30379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9bccfde25d9478ab1a0796c45112c2e: Generated new TSK 0
I20260928 16:55:16.236883 30379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:55:16.250427 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 30293
W20260928 16:55:16.251863 30355 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33001
W20260928 16:55:16.252799 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 30293: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:55:16.252986 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 30293
2026-09-28T16:55:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1962 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:55:16 dist-test-slave-wzcb krb5kdc[30391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:55:16 dist-test-slave-wzcb krb5kdc[30391](info): set up 2 sockets
Sep 28 16:55:16 dist-test-slave-wzcb krb5kdc[30391](info): commencing operation
krb5kdc: starting...
W20260928 16:55:17.302327 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:55:17 dist-test-slave-wzcb krb5kdc[30391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:55:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:55:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:55:17.607730 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46669
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:55:17.989256 30404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:17.989665 30404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:17.989827 30404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:17.997745 30404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:17.997910 30404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:17.998003 30404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:17.998096 30404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:17.998167 30404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:18.009107 30404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:18.010756 30404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:18.012362 30404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:18.026501 30409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:18.026520 30410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:18.026472 30412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:18.027388 30404 server_base.cc:1034] running on GCE node
I20260928 16:55:18.028482 30404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:18.030274 30404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:18.031508 30404 hybrid_clock.cc:648] HybridClock initialized: now 1790614518031451 us; error 95 us; skew 500 ppm
I20260928 16:55:18.031944 30404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:18 dist-test-slave-wzcb krb5kdc[30391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:18.035449 30404 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:55:18.037226 30404 webserver.cc:460] Webserver started at http://127.0.0.1:37787/ using document root <none> and password file <none>
I20260928 16:55:18.038071 30404 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:18.038188 30404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:18.038479 30404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:18.042016 30404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "456cd8ce6a6149d8afb3fc6d0b458a5f"
format_stamp: "Formatted at 2026-09-28 16:55:18 on dist-test-slave-wzcb"
server_key: "e865836b7537e3182772ed5f94df0f5a"
server_key_iv: "25f44c76f3d6ed90a69564ee831a48f5"
server_key_version: "encryptionkey@0"
I20260928 16:55:18.042882 30404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "456cd8ce6a6149d8afb3fc6d0b458a5f"
format_stamp: "Formatted at 2026-09-28 16:55:18 on dist-test-slave-wzcb"
server_key: "e865836b7537e3182772ed5f94df0f5a"
server_key_iv: "25f44c76f3d6ed90a69564ee831a48f5"
server_key_version: "encryptionkey@0"
I20260928 16:55:18.049052 30404 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 16:55:18.053468 30419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:18.055160 30404 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 16:55:18.055408 30404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "456cd8ce6a6149d8afb3fc6d0b458a5f"
format_stamp: "Formatted at 2026-09-28 16:55:18 on dist-test-slave-wzcb"
server_key: "e865836b7537e3182772ed5f94df0f5a"
server_key_iv: "25f44c76f3d6ed90a69564ee831a48f5"
server_key_version: "encryptionkey@0"
I20260928 16:55:18.055649 30404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:18.077723 30404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:18.082913 30404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:18.083202 30404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:18.103454 30404 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46669
I20260928 16:55:18.103495 30470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46669 every 8 connection(s)
I20260928 16:55:18.105046 30404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:55:18.109709 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 30404
I20260928 16:55:18.109809 30471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:18.109944 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:55:18.110562 9588 external_mini_cluster.cc:1428] Setting key c24fa9415f1dc9320d58c775bef52570
I20260928 16:55:18.127092 30471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 456cd8ce6a6149d8afb3fc6d0b458a5f: Bootstrap starting.
Sep 28 16:55:18 dist-test-slave-wzcb krb5kdc[30391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 16:55:18.131909 30471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 456cd8ce6a6149d8afb3fc6d0b458a5f: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:18.133816 30471 log.cc:824] T 00000000000000000000000000000000 P 456cd8ce6a6149d8afb3fc6d0b458a5f: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:18.138226 30471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 456cd8ce6a6149d8afb3fc6d0b458a5f: No bootstrap required, opened a new log
I20260928 16:55:18.139657 30474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:18.112686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48663 (local address 127.0.0.1:46669)
0928 16:55:18.113634 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:18.113653 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:18.113710 (+ 57us) server_negotiation.cc:407] Connection header received
0928 16:55:18.115656 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:18.115684 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:18.116214 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:18.116620 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:18.118700 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:18.120225 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:18.122256 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:18.123156 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:18.123387 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:18.128895 (+ 5508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:18.128916 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:18.128947 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:18.129023 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:18.133549 (+ 4526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:18.134747 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:18.134764 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:18.134772 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:18.135013 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:18.135694 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:18.135700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:18.135703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:18.136576 (+ 873us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:18.136922 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:18.137506 (+ 584us) server_negotiation.cc:299] Negotiation successful
0928 16:55:18.138052 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":425,"threads_started":1}
I20260928 16:55:18.144582 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:55:18.147908 30471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 456cd8ce6a6149d8afb3fc6d0b458a5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456cd8ce6a6149d8afb3fc6d0b458a5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46669 } }
I20260928 16:55:18.148358 30471 raft_consensus.cc:374] T 00000000000000000000000000000000 P 456cd8ce6a6149d8afb3fc6d0b458a5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:18.148476 30471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 456cd8ce6a6149d8afb3fc6d0b458a5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 456cd8ce6a6149d8afb3fc6d0b458a5f, State: Initialized, Role: FOLLOWER
I20260928 16:55:18.149248 30471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 456cd8ce6a6149d8afb3fc6d0b458a5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456cd8ce6a6149d8afb3fc6d0b458a5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46669 } }
I20260928 16:55:18.149605 30471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 456cd8ce6a6149d8afb3fc6d0b458a5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:55:18.149775 30471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 456cd8ce6a6149d8afb3fc6d0b458a5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:55:18.149984 30471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 456cd8ce6a6149d8afb3fc6d0b458a5f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:18.152551 30471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 456cd8ce6a6149d8afb3fc6d0b458a5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456cd8ce6a6149d8afb3fc6d0b458a5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46669 } }
I20260928 16:55:18.153169 30471 leader_election.cc:302] T 00000000000000000000000000000000 P 456cd8ce6a6149d8afb3fc6d0b458a5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 456cd8ce6a6149d8afb3fc6d0b458a5f; no voters:
I20260928 16:55:18.153940 30471 leader_election.cc:288] T 00000000000000000000000000000000 P 456cd8ce6a6149d8afb3fc6d0b458a5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:55:18.154042 30481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 456cd8ce6a6149d8afb3fc6d0b458a5f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:55:18.155663 30481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 456cd8ce6a6149d8afb3fc6d0b458a5f [term 1 LEADER]: Becoming Leader. State: Replica: 456cd8ce6a6149d8afb3fc6d0b458a5f, State: Running, Role: LEADER
I20260928 16:55:18.156417 30481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 456cd8ce6a6149d8afb3fc6d0b458a5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456cd8ce6a6149d8afb3fc6d0b458a5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46669 } }
I20260928 16:55:18.156796 30471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 456cd8ce6a6149d8afb3fc6d0b458a5f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:55:18.165457 30474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:18.149362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37047 (local address 127.0.0.1:46669)
0928 16:55:18.149545 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:18.149551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:18.150157 (+ 606us) server_negotiation.cc:407] Connection header received
0928 16:55:18.150379 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:18.150383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:18.150457 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:18.150546 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:18.152285 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:18.153089 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:18.154762 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:18.155415 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:18.155565 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:18.158907 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:18.158916 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:18.158922 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:18.158971 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:18.162197 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:18.163137 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:18.163144 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:18.163148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:18.163298 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:18.163791 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:18.163797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:18.163801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:18.164156 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:18.164436 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:18.164788 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 16:55:18.165035 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:55:18.168604 30483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 456cd8ce6a6149d8afb3fc6d0b458a5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 456cd8ce6a6149d8afb3fc6d0b458a5f. Latest consensus state: current_term: 1 leader_uuid: "456cd8ce6a6149d8afb3fc6d0b458a5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456cd8ce6a6149d8afb3fc6d0b458a5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46669 } } }
I20260928 16:55:18.169188 30483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 456cd8ce6a6149d8afb3fc6d0b458a5f [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:18.169243 30482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 456cd8ce6a6149d8afb3fc6d0b458a5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "456cd8ce6a6149d8afb3fc6d0b458a5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456cd8ce6a6149d8afb3fc6d0b458a5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46669 } } }
I20260928 16:55:18.169548 30482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 456cd8ce6a6149d8afb3fc6d0b458a5f [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:18.172096 30490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:55:18.178892 30490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:55:18.190637 30490 catalog_manager.cc:1347] Generated new cluster ID: 0f1d603f3ac946e5b9401340f0622e21
I20260928 16:55:18.190797 30490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:55:18.199354 30490 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:55:18.201460 30490 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:55:18.213465 30490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 456cd8ce6a6149d8afb3fc6d0b458a5f: Generated new TSK 0
I20260928 16:55:18.214831 30490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:55:18.228933 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 30404
I20260928 16:55:18.230332 30466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:55:18.324673 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 30404
2026-09-28T16:55:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2071 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:55:18 dist-test-slave-wzcb krb5kdc[30504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:55:18 dist-test-slave-wzcb krb5kdc[30504](info): set up 2 sockets
Sep 28 16:55:18 dist-test-slave-wzcb krb5kdc[30504](info): commencing operation
krb5kdc: starting...
W20260928 16:55:19.390561 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:55:19 dist-test-slave-wzcb krb5kdc[30504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:55:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:55:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:55:19.699481 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38139
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:55:20.077730 30517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:20.078106 30517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:20.078262 30517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:20.086126 30517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:20.086292 30517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:20.086383 30517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:20.086479 30517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:20.086539 30517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:20.097450 30517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:20.099030 30517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:20.100694 30517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:20.114475 30522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:20.114517 30523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:20.114467 30525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:20.115274 30517 server_base.cc:1034] running on GCE node
I20260928 16:55:20.116343 30517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:20.118234 30517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:20.119451 30517 hybrid_clock.cc:648] HybridClock initialized: now 1790614520119385 us; error 49 us; skew 500 ppm
I20260928 16:55:20.119902 30517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:20 dist-test-slave-wzcb krb5kdc[30504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:20.123759 30517 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:55:20.125831 30517 webserver.cc:460] Webserver started at http://127.0.0.1:36847/ using document root <none> and password file <none>
I20260928 16:55:20.126748 30517 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:20.126920 30517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:20.127383 30517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:20.131215 30517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "c06b8bd94db1486ab96a474f5c9f596d"
format_stamp: "Formatted at 2026-09-28 16:55:20 on dist-test-slave-wzcb"
server_key: "ac9e1a35fef5ccf4f6485f094e5e0c9f"
server_key_iv: "6ddf086885645b0496a079aa223ecf06"
server_key_version: "encryptionkey@0"
I20260928 16:55:20.132351 30517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "c06b8bd94db1486ab96a474f5c9f596d"
format_stamp: "Formatted at 2026-09-28 16:55:20 on dist-test-slave-wzcb"
server_key: "ac9e1a35fef5ccf4f6485f094e5e0c9f"
server_key_iv: "6ddf086885645b0496a079aa223ecf06"
server_key_version: "encryptionkey@0"
I20260928 16:55:20.139137 30517 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260928 16:55:20.143886 30532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:20.145756 30517 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:55:20.145994 30517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "c06b8bd94db1486ab96a474f5c9f596d"
format_stamp: "Formatted at 2026-09-28 16:55:20 on dist-test-slave-wzcb"
server_key: "ac9e1a35fef5ccf4f6485f094e5e0c9f"
server_key_iv: "6ddf086885645b0496a079aa223ecf06"
server_key_version: "encryptionkey@0"
I20260928 16:55:20.146224 30517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:20.167147 30517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:20.172292 30517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:20.172622 30517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:20.192922 30517 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38139
I20260928 16:55:20.192971 30583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38139 every 8 connection(s)
I20260928 16:55:20.194519 30517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:55:20.199149 30584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:20.202045 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 30517
I20260928 16:55:20.202271 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:55:20.202836 9588 external_mini_cluster.cc:1428] Setting key 86b4301fd4dfe6dedc627523647426b5
I20260928 16:55:20.213884 30584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c06b8bd94db1486ab96a474f5c9f596d: Bootstrap starting.
I20260928 16:55:20.218394 30584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c06b8bd94db1486ab96a474f5c9f596d: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:20.219802 30584 log.cc:824] T 00000000000000000000000000000000 P c06b8bd94db1486ab96a474f5c9f596d: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:20.219970 30587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:20.205326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33253 (local address 127.0.0.1:38139)
0928 16:55:20.206380 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:20.206397 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:20.206459 (+ 62us) server_negotiation.cc:407] Connection header received
0928 16:55:20.208690 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:20.208738 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:20.209339 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:20.209863 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:20.210643 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:20.212371 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:20.214411 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:20.215244 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:20.215507 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:20.216911 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:20.216921 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:20.216939 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:20.217567 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:20.217664 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:20.217694 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:20.217811 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:20.218302 (+ 491us) server_negotiation.cc:299] Negotiation successful
0928 16:55:20.218663 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":487,"threads_started":1}
I20260928 16:55:20.223440 30584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c06b8bd94db1486ab96a474f5c9f596d: No bootstrap required, opened a new log
I20260928 16:55:20.224829 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:55:20.232939 30584 raft_consensus.cc:348] T 00000000000000000000000000000000 P c06b8bd94db1486ab96a474f5c9f596d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c06b8bd94db1486ab96a474f5c9f596d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38139 } }
I20260928 16:55:20.233633 30584 raft_consensus.cc:374] T 00000000000000000000000000000000 P c06b8bd94db1486ab96a474f5c9f596d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:20.233777 30584 raft_consensus.cc:729] T 00000000000000000000000000000000 P c06b8bd94db1486ab96a474f5c9f596d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c06b8bd94db1486ab96a474f5c9f596d, State: Initialized, Role: FOLLOWER
I20260928 16:55:20.234643 30584 consensus_queue.cc:260] T 00000000000000000000000000000000 P c06b8bd94db1486ab96a474f5c9f596d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c06b8bd94db1486ab96a474f5c9f596d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38139 } }
I20260928 16:55:20.234935 30584 raft_consensus.cc:388] T 00000000000000000000000000000000 P c06b8bd94db1486ab96a474f5c9f596d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:55:20.235087 30584 raft_consensus.cc:482] T 00000000000000000000000000000000 P c06b8bd94db1486ab96a474f5c9f596d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:55:20.235266 30584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c06b8bd94db1486ab96a474f5c9f596d [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:20.237479 30587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:20.229882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51989 (local address 127.0.0.1:38139)
0928 16:55:20.230151 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:20.230160 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:20.230588 (+ 428us) server_negotiation.cc:407] Connection header received
0928 16:55:20.230793 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:20.230798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:20.230882 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:20.231002 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:20.231761 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:20.232554 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:20.234610 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:20.235163 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:20.235341 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:20.236572 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:20.236577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:20.236584 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:20.236607 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:20.236659 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:20.236677 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:20.236756 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:20.236967 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 16:55:20.237077 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260928 16:55:20.238564 30584 raft_consensus.cc:504] T 00000000000000000000000000000000 P c06b8bd94db1486ab96a474f5c9f596d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c06b8bd94db1486ab96a474f5c9f596d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38139 } }
I20260928 16:55:20.239203 30584 leader_election.cc:302] T 00000000000000000000000000000000 P c06b8bd94db1486ab96a474f5c9f596d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c06b8bd94db1486ab96a474f5c9f596d; no voters:
I20260928 16:55:20.239952 30584 leader_election.cc:288] T 00000000000000000000000000000000 P c06b8bd94db1486ab96a474f5c9f596d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:55:20.240097 30594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c06b8bd94db1486ab96a474f5c9f596d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:55:20.240820 30594 raft_consensus.cc:686] T 00000000000000000000000000000000 P c06b8bd94db1486ab96a474f5c9f596d [term 1 LEADER]: Becoming Leader. State: Replica: c06b8bd94db1486ab96a474f5c9f596d, State: Running, Role: LEADER
I20260928 16:55:20.241403 30594 consensus_queue.cc:237] T 00000000000000000000000000000000 P c06b8bd94db1486ab96a474f5c9f596d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c06b8bd94db1486ab96a474f5c9f596d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38139 } }
I20260928 16:55:20.242434 30584 sys_catalog.cc:564] T 00000000000000000000000000000000 P c06b8bd94db1486ab96a474f5c9f596d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:55:20.244958 30596 sys_catalog.cc:455] T 00000000000000000000000000000000 P c06b8bd94db1486ab96a474f5c9f596d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c06b8bd94db1486ab96a474f5c9f596d. Latest consensus state: current_term: 1 leader_uuid: "c06b8bd94db1486ab96a474f5c9f596d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c06b8bd94db1486ab96a474f5c9f596d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38139 } } }
I20260928 16:55:20.245807 30596 sys_catalog.cc:458] T 00000000000000000000000000000000 P c06b8bd94db1486ab96a474f5c9f596d [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:20.244942 30595 sys_catalog.cc:455] T 00000000000000000000000000000000 P c06b8bd94db1486ab96a474f5c9f596d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c06b8bd94db1486ab96a474f5c9f596d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c06b8bd94db1486ab96a474f5c9f596d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38139 } } }
I20260928 16:55:20.246337 30595 sys_catalog.cc:458] T 00000000000000000000000000000000 P c06b8bd94db1486ab96a474f5c9f596d [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:20.252660 30601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:55:20.258213 30601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:55:20.268680 30601 catalog_manager.cc:1347] Generated new cluster ID: ff16661ee9f04ba4a103d0d78804aa13
I20260928 16:55:20.268838 30601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:55:20.289834 30601 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:55:20.292132 30601 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:55:20.303167 30601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c06b8bd94db1486ab96a474f5c9f596d: Generated new TSK 0
I20260928 16:55:20.304538 30601 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:55:20.311003 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 30517
W20260928 16:55:20.312618 30579 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33253
W20260928 16:55:20.313537 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 30517: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:55:20.313690 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 30517
2026-09-28T16:55:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1989 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:55:20 dist-test-slave-wzcb krb5kdc[30615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:55:20 dist-test-slave-wzcb krb5kdc[30615](info): set up 2 sockets
Sep 28 16:55:20 dist-test-slave-wzcb krb5kdc[30615](info): commencing operation
krb5kdc: starting...
W20260928 16:55:21.370244 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:55:21 dist-test-slave-wzcb krb5kdc[30615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:55:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:55:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:55:21.683087 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41531
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:55:22.060770 30628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:22.061151 30628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:22.061262 30628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:22.069159 30628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:22.069326 30628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:22.069381 30628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:22.069448 30628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:22.069495 30628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:22.080363 30628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:22.081889 30628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:22.083467 30628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:22.097409 30636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:22.097447 30633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:22.097478 30634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:22.098366 30628 server_base.cc:1034] running on GCE node
I20260928 16:55:22.099426 30628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:22.101298 30628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:22.102535 30628 hybrid_clock.cc:648] HybridClock initialized: now 1790614522102455 us; error 82 us; skew 500 ppm
I20260928 16:55:22.102974 30628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:22 dist-test-slave-wzcb krb5kdc[30615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:22.106695 30628 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:55:22.108664 30628 webserver.cc:460] Webserver started at http://127.0.0.1:38695/ using document root <none> and password file <none>
I20260928 16:55:22.109547 30628 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:22.109692 30628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:22.110037 30628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:22.113752 30628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "d2b2627c321d4eb7bc4a9b2a94a8273e"
format_stamp: "Formatted at 2026-09-28 16:55:22 on dist-test-slave-wzcb"
server_key: "bb244f5b67d1f72d796c5b1efeef0b1a"
server_key_iv: "b2c1b4641504b8e9d9dee097054eb81e"
server_key_version: "encryptionkey@0"
I20260928 16:55:22.114751 30628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "d2b2627c321d4eb7bc4a9b2a94a8273e"
format_stamp: "Formatted at 2026-09-28 16:55:22 on dist-test-slave-wzcb"
server_key: "bb244f5b67d1f72d796c5b1efeef0b1a"
server_key_iv: "b2c1b4641504b8e9d9dee097054eb81e"
server_key_version: "encryptionkey@0"
I20260928 16:55:22.121279 30628 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 16:55:22.125917 30643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:22.127880 30628 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260928 16:55:22.128129 30628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "d2b2627c321d4eb7bc4a9b2a94a8273e"
format_stamp: "Formatted at 2026-09-28 16:55:22 on dist-test-slave-wzcb"
server_key: "bb244f5b67d1f72d796c5b1efeef0b1a"
server_key_iv: "b2c1b4641504b8e9d9dee097054eb81e"
server_key_version: "encryptionkey@0"
I20260928 16:55:22.128381 30628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:22.135632 30628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:22.140710 30628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:22.141057 30628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:22.161212 30628 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41531
I20260928 16:55:22.161267 30694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41531 every 8 connection(s)
I20260928 16:55:22.162865 30628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:55:22.165491 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 30628
I20260928 16:55:22.165679 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:55:22.166160 9588 external_mini_cluster.cc:1428] Setting key 910e65714dfbdd0753467134d4c52130
I20260928 16:55:22.167809 30695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:22.175531 30698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:22.168200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51249 (local address 127.0.0.1:41531)
0928 16:55:22.169087 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:22.169104 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:22.169153 (+ 49us) server_negotiation.cc:407] Connection header received
0928 16:55:22.170975 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:22.171006 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:22.171587 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:22.172182 (+ 595us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:22.173480 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:22.173487 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:22.173505 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:22.173528 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:22.173618 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:22.173647 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:22.173861 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:22.173986 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 16:55:22.174241 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":383,"threads_started":1}
I20260928 16:55:22.180502 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:55:22.182715 30695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2b2627c321d4eb7bc4a9b2a94a8273e: Bootstrap starting.
I20260928 16:55:22.187647 30695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2b2627c321d4eb7bc4a9b2a94a8273e: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:22.188302 30698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:22.185423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60527 (local address 127.0.0.1:41531)
0928 16:55:22.185642 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:22.185648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:22.185998 (+ 350us) server_negotiation.cc:407] Connection header received
0928 16:55:22.186228 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:22.186233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:22.186300 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:22.186391 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:22.187478 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:22.187482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:22.187486 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:22.187504 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:22.187556 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:22.187575 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:22.187676 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:22.187843 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 16:55:22.187934 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 16:55:22.189356 30695 log.cc:824] T 00000000000000000000000000000000 P d2b2627c321d4eb7bc4a9b2a94a8273e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:22.192948 30695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2b2627c321d4eb7bc4a9b2a94a8273e: No bootstrap required, opened a new log
I20260928 16:55:22.201421 30695 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2b2627c321d4eb7bc4a9b2a94a8273e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b2627c321d4eb7bc4a9b2a94a8273e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41531 } }
I20260928 16:55:22.201951 30695 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2b2627c321d4eb7bc4a9b2a94a8273e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:22.202078 30695 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2b2627c321d4eb7bc4a9b2a94a8273e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2b2627c321d4eb7bc4a9b2a94a8273e, State: Initialized, Role: FOLLOWER
I20260928 16:55:22.202793 30695 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2b2627c321d4eb7bc4a9b2a94a8273e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b2627c321d4eb7bc4a9b2a94a8273e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41531 } }
I20260928 16:55:22.203045 30695 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2b2627c321d4eb7bc4a9b2a94a8273e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:55:22.203202 30695 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2b2627c321d4eb7bc4a9b2a94a8273e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:55:22.203408 30695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2b2627c321d4eb7bc4a9b2a94a8273e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:22.205964 30695 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2b2627c321d4eb7bc4a9b2a94a8273e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b2627c321d4eb7bc4a9b2a94a8273e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41531 } }
I20260928 16:55:22.206630 30695 leader_election.cc:302] T 00000000000000000000000000000000 P d2b2627c321d4eb7bc4a9b2a94a8273e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2b2627c321d4eb7bc4a9b2a94a8273e; no voters:
I20260928 16:55:22.207361 30695 leader_election.cc:288] T 00000000000000000000000000000000 P d2b2627c321d4eb7bc4a9b2a94a8273e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:55:22.207543 30705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2b2627c321d4eb7bc4a9b2a94a8273e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:55:22.208282 30705 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2b2627c321d4eb7bc4a9b2a94a8273e [term 1 LEADER]: Becoming Leader. State: Replica: d2b2627c321d4eb7bc4a9b2a94a8273e, State: Running, Role: LEADER
I20260928 16:55:22.208935 30705 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2b2627c321d4eb7bc4a9b2a94a8273e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b2627c321d4eb7bc4a9b2a94a8273e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41531 } }
I20260928 16:55:22.209836 30695 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2b2627c321d4eb7bc4a9b2a94a8273e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:55:22.212118 30707 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2b2627c321d4eb7bc4a9b2a94a8273e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2b2627c321d4eb7bc4a9b2a94a8273e. Latest consensus state: current_term: 1 leader_uuid: "d2b2627c321d4eb7bc4a9b2a94a8273e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b2627c321d4eb7bc4a9b2a94a8273e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41531 } } }
I20260928 16:55:22.212704 30707 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2b2627c321d4eb7bc4a9b2a94a8273e [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:22.212399 30706 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2b2627c321d4eb7bc4a9b2a94a8273e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2b2627c321d4eb7bc4a9b2a94a8273e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b2627c321d4eb7bc4a9b2a94a8273e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41531 } } }
I20260928 16:55:22.213831 30706 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2b2627c321d4eb7bc4a9b2a94a8273e [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:22.214285 30713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:55:22.220427 30713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:55:22.232857 30713 catalog_manager.cc:1347] Generated new cluster ID: 6b81250cc9c7437b8382b99d6050dbb5
I20260928 16:55:22.232997 30713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:55:22.273028 30713 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:55:22.273552 30713 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:55:22.285341 30713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2b2627c321d4eb7bc4a9b2a94a8273e: Generated new TSK 0
I20260928 16:55:22.286656 30713 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:55:22.306046 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 30628
W20260928 16:55:22.307459 30690 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51249
W20260928 16:55:22.308279 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 30628: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:55:22.308462 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 30628
2026-09-28T16:55:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1993 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:55:22 dist-test-slave-wzcb krb5kdc[30726](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:55:22 dist-test-slave-wzcb krb5kdc[30726](info): set up 2 sockets
Sep 28 16:55:22 dist-test-slave-wzcb krb5kdc[30726](info): commencing operation
krb5kdc: starting...
W20260928 16:55:23.366254 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:55:23 dist-test-slave-wzcb krb5kdc[30726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:55:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:55:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:55:23.675617 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44073
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:55:24.061553 30739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:24.061967 30739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:24.062124 30739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:24.070169 30739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:24.070348 30739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:24.070442 30739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:24.070533 30739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:24.070600 30739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:24.081693 30739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:24.083269 30739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:24.084882 30739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:24.098507 30744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:24.098507 30745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:24.098706 30747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:24.099208 30739 server_base.cc:1034] running on GCE node
I20260928 16:55:24.100224 30739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:24.102010 30739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:24.103269 30739 hybrid_clock.cc:648] HybridClock initialized: now 1790614524103197 us; error 56 us; skew 500 ppm
I20260928 16:55:24.103772 30739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:24 dist-test-slave-wzcb krb5kdc[30726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:24.107556 30739 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:55:24.109419 30739 webserver.cc:460] Webserver started at http://127.0.0.1:41157/ using document root <none> and password file <none>
I20260928 16:55:24.110280 30739 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:24.110430 30739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:24.110762 30739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:24.114383 30739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "61db0f953cce499f948216eb3b247b0d"
format_stamp: "Formatted at 2026-09-28 16:55:24 on dist-test-slave-wzcb"
server_key: "6b890236049ec45670c30396279e44c1"
server_key_iv: "5e3888324a9d5555910561b7c77f84b8"
server_key_version: "encryptionkey@0"
I20260928 16:55:24.115341 30739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "61db0f953cce499f948216eb3b247b0d"
format_stamp: "Formatted at 2026-09-28 16:55:24 on dist-test-slave-wzcb"
server_key: "6b890236049ec45670c30396279e44c1"
server_key_iv: "5e3888324a9d5555910561b7c77f84b8"
server_key_version: "encryptionkey@0"
I20260928 16:55:24.121935 30739 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:55:24.126407 30754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:24.128311 30739 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 16:55:24.128533 30739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "61db0f953cce499f948216eb3b247b0d"
format_stamp: "Formatted at 2026-09-28 16:55:24 on dist-test-slave-wzcb"
server_key: "6b890236049ec45670c30396279e44c1"
server_key_iv: "5e3888324a9d5555910561b7c77f84b8"
server_key_version: "encryptionkey@0"
I20260928 16:55:24.128763 30739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:24.151221 30739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:24.161959 30739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:24.162322 30739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:24.183260 30739 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44073
I20260928 16:55:24.183329 30805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44073 every 8 connection(s)
I20260928 16:55:24.184959 30739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:55:24.188668 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 30739
I20260928 16:55:24.188874 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:55:24.189417 9588 external_mini_cluster.cc:1428] Setting key 41a3281c2eb4ee7c5ae929bc0db46eeb
I20260928 16:55:24.189565 30806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:24.206334 30806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61db0f953cce499f948216eb3b247b0d: Bootstrap starting.
Sep 28 16:55:24 dist-test-slave-wzcb krb5kdc[30726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 16:55:24.210815 30806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61db0f953cce499f948216eb3b247b0d: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:24.212558 30806 log.cc:824] T 00000000000000000000000000000000 P 61db0f953cce499f948216eb3b247b0d: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:24.217132 30806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61db0f953cce499f948216eb3b247b0d: No bootstrap required, opened a new log
I20260928 16:55:24.219358 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:24.191612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.222:40005 (local address 127.0.0.1:44073)
0928 16:55:24.192500 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:24.192515 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:24.192566 (+ 51us) server_negotiation.cc:407] Connection header received
0928 16:55:24.195029 (+ 2463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:24.195062 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:24.195693 (+ 631us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:24.196154 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:24.198255 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:24.199921 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:24.201927 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:24.202835 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:24.203044 (+ 209us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:24.208699 (+ 5655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:24.208712 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:24.208738 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:24.208808 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:24.213214 (+ 4406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:24.214678 (+ 1464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:24.214689 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:24.214700 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:24.214954 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:24.215561 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:24.215566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:24.215569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:24.216388 (+ 819us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:24.216734 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:24.217285 (+ 551us) server_negotiation.cc:299] Negotiation successful
0928 16:55:24.217720 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":387,"threads_started":1}
I20260928 16:55:24.224471 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:55:24.227437 30806 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61db0f953cce499f948216eb3b247b0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61db0f953cce499f948216eb3b247b0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44073 } }
I20260928 16:55:24.228098 30806 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61db0f953cce499f948216eb3b247b0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:24.228235 30806 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61db0f953cce499f948216eb3b247b0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61db0f953cce499f948216eb3b247b0d, State: Initialized, Role: FOLLOWER
I20260928 16:55:24.229138 30806 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61db0f953cce499f948216eb3b247b0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61db0f953cce499f948216eb3b247b0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44073 } }
I20260928 16:55:24.229477 30806 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61db0f953cce499f948216eb3b247b0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:55:24.229648 30806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61db0f953cce499f948216eb3b247b0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:55:24.229862 30806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61db0f953cce499f948216eb3b247b0d [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:24.233484 30806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61db0f953cce499f948216eb3b247b0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61db0f953cce499f948216eb3b247b0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44073 } }
I20260928 16:55:24.234338 30806 leader_election.cc:302] T 00000000000000000000000000000000 P 61db0f953cce499f948216eb3b247b0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61db0f953cce499f948216eb3b247b0d; no voters:
I20260928 16:55:24.235137 30806 leader_election.cc:288] T 00000000000000000000000000000000 P 61db0f953cce499f948216eb3b247b0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:55:24.235270 30816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61db0f953cce499f948216eb3b247b0d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:55:24.237651 30806 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61db0f953cce499f948216eb3b247b0d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:55:24.239629 30816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61db0f953cce499f948216eb3b247b0d [term 1 LEADER]: Becoming Leader. State: Replica: 61db0f953cce499f948216eb3b247b0d, State: Running, Role: LEADER
I20260928 16:55:24.240541 30816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61db0f953cce499f948216eb3b247b0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61db0f953cce499f948216eb3b247b0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44073 } }
I20260928 16:55:24.244360 30817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61db0f953cce499f948216eb3b247b0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61db0f953cce499f948216eb3b247b0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61db0f953cce499f948216eb3b247b0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44073 } } }
I20260928 16:55:24.244905 30817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61db0f953cce499f948216eb3b247b0d [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:24.247576 30818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61db0f953cce499f948216eb3b247b0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61db0f953cce499f948216eb3b247b0d. Latest consensus state: current_term: 1 leader_uuid: "61db0f953cce499f948216eb3b247b0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61db0f953cce499f948216eb3b247b0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44073 } } }
I20260928 16:55:24.248009 30818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61db0f953cce499f948216eb3b247b0d [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:24.249333 30825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:55:24.251135 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:24.230253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.222:40425 (local address 127.0.0.1:44073)
0928 16:55:24.230568 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:24.230576 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:24.231210 (+ 634us) server_negotiation.cc:407] Connection header received
0928 16:55:24.231486 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:24.231490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:24.231574 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:24.231681 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:24.233916 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:24.235056 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:24.237233 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:24.237950 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:24.238095 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:24.241916 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:24.241925 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:24.241932 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:24.241995 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:24.247104 (+ 5109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:24.248434 (+ 1330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:24.248440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:24.248444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:24.248615 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:24.249287 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:24.249293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:24.249297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:24.249799 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:24.250109 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:24.250551 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 16:55:24.250756 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260928 16:55:24.255743 30825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:55:24.268929 30825 catalog_manager.cc:1347] Generated new cluster ID: dee75242c38a4cd5a8d8df6235d68c11
I20260928 16:55:24.269187 30825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:55:24.279594 30825 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:55:24.281157 30825 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:55:24.292872 30825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61db0f953cce499f948216eb3b247b0d: Generated new TSK 0
I20260928 16:55:24.294101 30825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:55:24.305431 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 30739
I20260928 16:55:24.306866 30801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:55:24.403265 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 30739
W20260928 16:55:24.420809 30724 connection.cc:535] client connection to 127.0.0.1:44073 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44073): Connection reset by peer (error 104)
2026-09-28T16:55:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2094 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:55:24 dist-test-slave-wzcb krb5kdc[30838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:55:24 dist-test-slave-wzcb krb5kdc[30838](info): set up 2 sockets
Sep 28 16:55:24 dist-test-slave-wzcb krb5kdc[30838](info): commencing operation
krb5kdc: starting...
W20260928 16:55:25.454054 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:55:25 dist-test-slave-wzcb krb5kdc[30838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:55:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:55:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:55:25.758802 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40681
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:55:26.138350 30851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:26.138698 30851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:26.138855 30851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:26.146588 30851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:26.146733 30851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:26.146790 30851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:26.146881 30851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:26.146957 30851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:26.157725 30851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:26.159260 30851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:26.160897 30851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:26.174242 30856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:26.174222 30857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:26.174181 30859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:26.175215 30851 server_base.cc:1034] running on GCE node
I20260928 16:55:26.176213 30851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:26.177939 30851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:26.179188 30851 hybrid_clock.cc:648] HybridClock initialized: now 1790614526179126 us; error 52 us; skew 500 ppm
I20260928 16:55:26.179615 30851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:26 dist-test-slave-wzcb krb5kdc[30838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:26.183038 30851 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:55:26.184890 30851 webserver.cc:460] Webserver started at http://127.0.0.1:43447/ using document root <none> and password file <none>
I20260928 16:55:26.185724 30851 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:26.185863 30851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:26.186197 30851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:26.189770 30851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "a6692ad6da1042a6b30bf9674b73bb1d"
format_stamp: "Formatted at 2026-09-28 16:55:26 on dist-test-slave-wzcb"
server_key: "153b98eb6792a2e98147700629b72e00"
server_key_iv: "4768f399f725c67f0f406323b7555c85"
server_key_version: "encryptionkey@0"
I20260928 16:55:26.190685 30851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "a6692ad6da1042a6b30bf9674b73bb1d"
format_stamp: "Formatted at 2026-09-28 16:55:26 on dist-test-slave-wzcb"
server_key: "153b98eb6792a2e98147700629b72e00"
server_key_iv: "4768f399f725c67f0f406323b7555c85"
server_key_version: "encryptionkey@0"
I20260928 16:55:26.196875 30851 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 16:55:26.200858 30866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:26.202507 30851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:55:26.202703 30851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "a6692ad6da1042a6b30bf9674b73bb1d"
format_stamp: "Formatted at 2026-09-28 16:55:26 on dist-test-slave-wzcb"
server_key: "153b98eb6792a2e98147700629b72e00"
server_key_iv: "4768f399f725c67f0f406323b7555c85"
server_key_version: "encryptionkey@0"
I20260928 16:55:26.202926 30851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:26.239094 30851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:26.244127 30851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:26.244452 30851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:26.263823 30851 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40681
I20260928 16:55:26.263857 30917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40681 every 8 connection(s)
I20260928 16:55:26.265506 30851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:55:26.270066 30918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:26.270838 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 30851
I20260928 16:55:26.271035 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:55:26.271682 9588 external_mini_cluster.cc:1428] Setting key 3f11b2c14db888c3ab6d5a2c039d042a
I20260928 16:55:26.284518 30918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6692ad6da1042a6b30bf9674b73bb1d: Bootstrap starting.
I20260928 16:55:26.287190 30921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:26.273863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.222:57019 (local address 127.0.0.1:40681)
0928 16:55:26.274755 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:26.274787 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:26.274844 (+ 57us) server_negotiation.cc:407] Connection header received
0928 16:55:26.277254 (+ 2410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:26.277285 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:26.277855 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:26.278321 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:26.278977 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:26.280427 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:26.281972 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:26.282836 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:26.283056 (+ 220us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:26.284510 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:26.284520 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:26.284537 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:26.285194 (+ 657us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:26.285287 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:26.285319 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:26.285470 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:26.285815 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 16:55:26.286019 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":390,"threads_started":1}
I20260928 16:55:26.288335 30918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6692ad6da1042a6b30bf9674b73bb1d: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:26.289875 30918 log.cc:824] T 00000000000000000000000000000000 P a6692ad6da1042a6b30bf9674b73bb1d: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:26.291999 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:55:26.293923 30918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6692ad6da1042a6b30bf9674b73bb1d: No bootstrap required, opened a new log
I20260928 16:55:26.305474 30921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:26.297654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.222:44471 (local address 127.0.0.1:40681)
0928 16:55:26.297873 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:26.297881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:26.298469 (+ 588us) server_negotiation.cc:407] Connection header received
0928 16:55:26.298634 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:26.298640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:26.298737 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:26.298873 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:26.299434 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:26.300643 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:26.302528 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:26.303265 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:26.303503 (+ 238us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:26.304694 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:26.304702 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:26.304707 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:26.304734 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:26.304793 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:26.304815 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:26.304935 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:26.305068 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 16:55:26.305131 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:55:26.305446 30918 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6692ad6da1042a6b30bf9674b73bb1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6692ad6da1042a6b30bf9674b73bb1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40681 } }
I20260928 16:55:26.306035 30918 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6692ad6da1042a6b30bf9674b73bb1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:26.306174 30918 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6692ad6da1042a6b30bf9674b73bb1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6692ad6da1042a6b30bf9674b73bb1d, State: Initialized, Role: FOLLOWER
I20260928 16:55:26.306921 30918 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6692ad6da1042a6b30bf9674b73bb1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6692ad6da1042a6b30bf9674b73bb1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40681 } }
I20260928 16:55:26.307240 30918 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6692ad6da1042a6b30bf9674b73bb1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:55:26.307430 30918 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6692ad6da1042a6b30bf9674b73bb1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:55:26.307610 30918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6692ad6da1042a6b30bf9674b73bb1d [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:26.310099 30918 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6692ad6da1042a6b30bf9674b73bb1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6692ad6da1042a6b30bf9674b73bb1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40681 } }
I20260928 16:55:26.310705 30918 leader_election.cc:302] T 00000000000000000000000000000000 P a6692ad6da1042a6b30bf9674b73bb1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6692ad6da1042a6b30bf9674b73bb1d; no voters:
I20260928 16:55:26.311386 30918 leader_election.cc:288] T 00000000000000000000000000000000 P a6692ad6da1042a6b30bf9674b73bb1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:55:26.311565 30928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6692ad6da1042a6b30bf9674b73bb1d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:55:26.313181 30928 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6692ad6da1042a6b30bf9674b73bb1d [term 1 LEADER]: Becoming Leader. State: Replica: a6692ad6da1042a6b30bf9674b73bb1d, State: Running, Role: LEADER
I20260928 16:55:26.313782 30928 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6692ad6da1042a6b30bf9674b73bb1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6692ad6da1042a6b30bf9674b73bb1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40681 } }
I20260928 16:55:26.313951 30918 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6692ad6da1042a6b30bf9674b73bb1d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:55:26.316982 30929 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6692ad6da1042a6b30bf9674b73bb1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6692ad6da1042a6b30bf9674b73bb1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6692ad6da1042a6b30bf9674b73bb1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40681 } } }
I20260928 16:55:26.316977 30930 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6692ad6da1042a6b30bf9674b73bb1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6692ad6da1042a6b30bf9674b73bb1d. Latest consensus state: current_term: 1 leader_uuid: "a6692ad6da1042a6b30bf9674b73bb1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6692ad6da1042a6b30bf9674b73bb1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40681 } } }
I20260928 16:55:26.317509 30929 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6692ad6da1042a6b30bf9674b73bb1d [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:26.317575 30930 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6692ad6da1042a6b30bf9674b73bb1d [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:26.318455 30937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:55:26.327440 30937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:55:26.339423 30937 catalog_manager.cc:1347] Generated new cluster ID: 10f7e8bf30ae4b0bba9ab0da3eee6608
I20260928 16:55:26.339562 30937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:55:26.360929 30937 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:55:26.362489 30937 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:55:26.369329 30937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6692ad6da1042a6b30bf9674b73bb1d: Generated new TSK 0
I20260928 16:55:26.370628 30937 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:55:26.382153 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 30851
W20260928 16:55:26.383643 30913 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.222:57019
W20260928 16:55:26.384677 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 30851: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:55:26.384893 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 30851
W20260928 16:55:26.401891 30836 connection.cc:535] client connection to 127.0.0.1:40681 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40681): Connection reset by peer (error 104)
2026-09-28T16:55:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1981 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:55:26 dist-test-slave-wzcb krb5kdc[30949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:55:26 dist-test-slave-wzcb krb5kdc[30949](info): set up 2 sockets
Sep 28 16:55:26 dist-test-slave-wzcb krb5kdc[30949](info): commencing operation
krb5kdc: starting...
W20260928 16:55:27.434365 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:55:27 dist-test-slave-wzcb krb5kdc[30949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:55:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:55:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:55:27.747681 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40795
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:55:28.126965 30962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:28.127377 30962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:28.127528 30962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:28.135468 30962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:28.135663 30962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:28.135757 30962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:28.135844 30962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:28.135910 30962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:28.146751 30962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:28.148355 30962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:28.149907 30962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:28.163658 30967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:28.163712 30970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:28.163590 30968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:28.164842 30962 server_base.cc:1034] running on GCE node
I20260928 16:55:28.165841 30962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:28.167639 30962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:28.168913 30962 hybrid_clock.cc:648] HybridClock initialized: now 1790614528168820 us; error 76 us; skew 500 ppm
I20260928 16:55:28.169342 30962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:28 dist-test-slave-wzcb krb5kdc[30949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:28.173414 30962 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:55:28.175457 30962 webserver.cc:460] Webserver started at http://127.0.0.1:33155/ using document root <none> and password file <none>
I20260928 16:55:28.176345 30962 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:28.176510 30962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:28.176887 30962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:28.180599 30962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "a42662b8b73e48bcb62c2993d87d7e7e"
format_stamp: "Formatted at 2026-09-28 16:55:28 on dist-test-slave-wzcb"
server_key: "d19034d7852dfd6ca0934b0aaf2a7a8e"
server_key_iv: "d38142127c0cf1f9015a22f19263d9c1"
server_key_version: "encryptionkey@0"
I20260928 16:55:28.181550 30962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "a42662b8b73e48bcb62c2993d87d7e7e"
format_stamp: "Formatted at 2026-09-28 16:55:28 on dist-test-slave-wzcb"
server_key: "d19034d7852dfd6ca0934b0aaf2a7a8e"
server_key_iv: "d38142127c0cf1f9015a22f19263d9c1"
server_key_version: "encryptionkey@0"
I20260928 16:55:28.187873 30962 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 16:55:28.192216 30977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:28.194114 30962 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:55:28.194386 30962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "a42662b8b73e48bcb62c2993d87d7e7e"
format_stamp: "Formatted at 2026-09-28 16:55:28 on dist-test-slave-wzcb"
server_key: "d19034d7852dfd6ca0934b0aaf2a7a8e"
server_key_iv: "d38142127c0cf1f9015a22f19263d9c1"
server_key_version: "encryptionkey@0"
I20260928 16:55:28.194649 30962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:28.201874 30962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:28.207139 30962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:28.207525 30962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:28.226951 30962 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40795
I20260928 16:55:28.226992 31028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40795 every 8 connection(s)
I20260928 16:55:28.228605 30962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:55:28.229604 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 30962
I20260928 16:55:28.229802 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:55:28.230278 9588 external_mini_cluster.cc:1428] Setting key fbba1efdaf07d7468ab96120850050a4
I20260928 16:55:28.234469 31029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:28.239130 31032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:28.232283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.222:50047 (local address 127.0.0.1:40795)
0928 16:55:28.233161 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:28.233175 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:28.233224 (+ 49us) server_negotiation.cc:407] Connection header received
0928 16:55:28.235053 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:28.235081 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:28.235530 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:28.236086 (+ 556us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:28.237238 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:28.237245 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:28.237262 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:28.237279 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:28.237388 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:28.237417 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:28.237612 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:28.237733 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 16:55:28.237990 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":361,"threads_started":1}
I20260928 16:55:28.243918 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:55:28.248097 31029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a42662b8b73e48bcb62c2993d87d7e7e: Bootstrap starting.
I20260928 16:55:28.251521 31032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:28.248629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.222:56507 (local address 127.0.0.1:40795)
0928 16:55:28.248869 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:28.248878 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:28.249443 (+ 565us) server_negotiation.cc:407] Connection header received
0928 16:55:28.249592 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:28.249598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:28.249669 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:28.249811 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:28.250803 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:28.250809 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:28.250814 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:28.250834 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:28.250887 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:28.250908 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:28.251032 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:28.251101 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 16:55:28.251169 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:55:28.252385 31029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a42662b8b73e48bcb62c2993d87d7e7e: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:28.253790 31029 log.cc:824] T 00000000000000000000000000000000 P a42662b8b73e48bcb62c2993d87d7e7e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:28.257328 31029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a42662b8b73e48bcb62c2993d87d7e7e: No bootstrap required, opened a new log
I20260928 16:55:28.265506 31029 raft_consensus.cc:348] T 00000000000000000000000000000000 P a42662b8b73e48bcb62c2993d87d7e7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42662b8b73e48bcb62c2993d87d7e7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40795 } }
I20260928 16:55:28.265960 31029 raft_consensus.cc:374] T 00000000000000000000000000000000 P a42662b8b73e48bcb62c2993d87d7e7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:28.266052 31029 raft_consensus.cc:729] T 00000000000000000000000000000000 P a42662b8b73e48bcb62c2993d87d7e7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a42662b8b73e48bcb62c2993d87d7e7e, State: Initialized, Role: FOLLOWER
I20260928 16:55:28.266659 31029 consensus_queue.cc:260] T 00000000000000000000000000000000 P a42662b8b73e48bcb62c2993d87d7e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42662b8b73e48bcb62c2993d87d7e7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40795 } }
I20260928 16:55:28.266893 31029 raft_consensus.cc:388] T 00000000000000000000000000000000 P a42662b8b73e48bcb62c2993d87d7e7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:55:28.267071 31029 raft_consensus.cc:482] T 00000000000000000000000000000000 P a42662b8b73e48bcb62c2993d87d7e7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:55:28.267269 31029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a42662b8b73e48bcb62c2993d87d7e7e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:28.269855 31029 raft_consensus.cc:504] T 00000000000000000000000000000000 P a42662b8b73e48bcb62c2993d87d7e7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42662b8b73e48bcb62c2993d87d7e7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40795 } }
I20260928 16:55:28.270440 31029 leader_election.cc:302] T 00000000000000000000000000000000 P a42662b8b73e48bcb62c2993d87d7e7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a42662b8b73e48bcb62c2993d87d7e7e; no voters:
I20260928 16:55:28.271072 31029 leader_election.cc:288] T 00000000000000000000000000000000 P a42662b8b73e48bcb62c2993d87d7e7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:55:28.271239 31039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a42662b8b73e48bcb62c2993d87d7e7e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:55:28.271981 31039 raft_consensus.cc:686] T 00000000000000000000000000000000 P a42662b8b73e48bcb62c2993d87d7e7e [term 1 LEADER]: Becoming Leader. State: Replica: a42662b8b73e48bcb62c2993d87d7e7e, State: Running, Role: LEADER
I20260928 16:55:28.272604 31039 consensus_queue.cc:237] T 00000000000000000000000000000000 P a42662b8b73e48bcb62c2993d87d7e7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42662b8b73e48bcb62c2993d87d7e7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40795 } }
I20260928 16:55:28.273408 31029 sys_catalog.cc:564] T 00000000000000000000000000000000 P a42662b8b73e48bcb62c2993d87d7e7e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:55:28.275665 31040 sys_catalog.cc:455] T 00000000000000000000000000000000 P a42662b8b73e48bcb62c2993d87d7e7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a42662b8b73e48bcb62c2993d87d7e7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42662b8b73e48bcb62c2993d87d7e7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40795 } } }
I20260928 16:55:28.276103 31040 sys_catalog.cc:458] T 00000000000000000000000000000000 P a42662b8b73e48bcb62c2993d87d7e7e [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:28.276578 31041 sys_catalog.cc:455] T 00000000000000000000000000000000 P a42662b8b73e48bcb62c2993d87d7e7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a42662b8b73e48bcb62c2993d87d7e7e. Latest consensus state: current_term: 1 leader_uuid: "a42662b8b73e48bcb62c2993d87d7e7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42662b8b73e48bcb62c2993d87d7e7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40795 } } }
I20260928 16:55:28.276947 31041 sys_catalog.cc:458] T 00000000000000000000000000000000 P a42662b8b73e48bcb62c2993d87d7e7e [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:28.277101 31044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:55:28.284241 31044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:55:28.296007 31044 catalog_manager.cc:1347] Generated new cluster ID: ecc7344afd1241459c3fa01142098c45
I20260928 16:55:28.296159 31044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:55:28.311355 31044 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:55:28.311888 31044 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:55:28.321080 31044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a42662b8b73e48bcb62c2993d87d7e7e: Generated new TSK 0
I20260928 16:55:28.322149 31044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:55:28.330220 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 30962
W20260928 16:55:28.331993 31024 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.222:50047
W20260928 16:55:28.332914 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 30962: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:55:28.333105 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 30962
2026-09-28T16:55:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1948 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:55:28 dist-test-slave-wzcb krb5kdc[31060](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:55:28 dist-test-slave-wzcb krb5kdc[31060](info): set up 2 sockets
Sep 28 16:55:28 dist-test-slave-wzcb krb5kdc[31060](info): commencing operation
krb5kdc: starting...
W20260928 16:55:29.390189 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.007s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:55:29 dist-test-slave-wzcb krb5kdc[31060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:55:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:55:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:55:29.699244 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32967
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:55:30.080041 31073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:30.080410 31073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:30.080521 31073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:30.089085 31073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:30.089244 31073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:30.089294 31073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:30.089357 31073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:30.089402 31073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:30.100390 31073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32967
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:30.101974 31073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:30.103533 31073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:30.116890 31078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:30.116873 31079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:30.116873 31081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:30.118098 31073 server_base.cc:1034] running on GCE node
I20260928 16:55:30.119115 31073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:30.120962 31073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:30.122217 31073 hybrid_clock.cc:648] HybridClock initialized: now 1790614530122150 us; error 64 us; skew 500 ppm
I20260928 16:55:30.122658 31073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:30 dist-test-slave-wzcb krb5kdc[31060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:30.126338 31073 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:55:30.128274 31073 webserver.cc:460] Webserver started at http://127.0.0.1:36989/ using document root <none> and password file <none>
I20260928 16:55:30.129340 31073 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:30.129500 31073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:30.129899 31073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:30.133575 31073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "6624d5c725814ceb93acb2f41c69d1f8"
format_stamp: "Formatted at 2026-09-28 16:55:30 on dist-test-slave-wzcb"
server_key: "5d6ce1f401660e8722c9ec00a7e48466"
server_key_iv: "ffad217d6abad4d931a913ee6ae885f1"
server_key_version: "encryptionkey@0"
I20260928 16:55:30.134514 31073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "6624d5c725814ceb93acb2f41c69d1f8"
format_stamp: "Formatted at 2026-09-28 16:55:30 on dist-test-slave-wzcb"
server_key: "5d6ce1f401660e8722c9ec00a7e48466"
server_key_iv: "ffad217d6abad4d931a913ee6ae885f1"
server_key_version: "encryptionkey@0"
I20260928 16:55:30.141680 31073 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260928 16:55:30.146358 31088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:30.148201 31073 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 16:55:30.148479 31073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "6624d5c725814ceb93acb2f41c69d1f8"
format_stamp: "Formatted at 2026-09-28 16:55:30 on dist-test-slave-wzcb"
server_key: "5d6ce1f401660e8722c9ec00a7e48466"
server_key_iv: "ffad217d6abad4d931a913ee6ae885f1"
server_key_version: "encryptionkey@0"
I20260928 16:55:30.148728 31073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:30.186430 31073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:30.191541 31073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:30.191869 31073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:30.212211 31073 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32967
I20260928 16:55:30.212249 31139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32967 every 8 connection(s)
I20260928 16:55:30.213774 31073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:55:30.218349 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:30.222005 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 31073
I20260928 16:55:30.222221 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:55:30.222805 9588 external_mini_cluster.cc:1428] Setting key 7746cbde2b4c24ad08e3c62a8dceae4c
I20260928 16:55:30.231781 31140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6624d5c725814ceb93acb2f41c69d1f8: Bootstrap starting.
I20260928 16:55:30.235443 31140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6624d5c725814ceb93acb2f41c69d1f8: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:30.236630 31140 log.cc:824] T 00000000000000000000000000000000 P 6624d5c725814ceb93acb2f41c69d1f8: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:55:30 dist-test-slave-wzcb krb5kdc[31060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 16:55:30.240337 31140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6624d5c725814ceb93acb2f41c69d1f8: No bootstrap required, opened a new log
I20260928 16:55:30.249708 31143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:30.225062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.222:50239 (local address 127.0.0.1:32967)
0928 16:55:30.225961 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:30.225980 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:30.226043 (+ 63us) server_negotiation.cc:407] Connection header received
0928 16:55:30.228065 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:30.228092 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:30.228615 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:30.229068 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:30.230683 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:30.232235 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:30.233673 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:30.234464 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:30.234659 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:30.239290 (+ 4631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:30.239337 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:30.239363 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:30.239424 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:30.243871 (+ 4447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:30.244967 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:30.244978 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:30.244989 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:30.245244 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:30.245706 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:30.245711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:30.245714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:30.246583 (+ 869us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:30.246953 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:30.247562 (+ 609us) server_negotiation.cc:299] Negotiation successful
0928 16:55:30.247996 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":373,"threads_started":1}
I20260928 16:55:30.251471 31140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6624d5c725814ceb93acb2f41c69d1f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6624d5c725814ceb93acb2f41c69d1f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32967 } }
I20260928 16:55:30.252038 31140 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6624d5c725814ceb93acb2f41c69d1f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:30.252156 31140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6624d5c725814ceb93acb2f41c69d1f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6624d5c725814ceb93acb2f41c69d1f8, State: Initialized, Role: FOLLOWER
I20260928 16:55:30.252880 31140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6624d5c725814ceb93acb2f41c69d1f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6624d5c725814ceb93acb2f41c69d1f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32967 } }
I20260928 16:55:30.253185 31140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6624d5c725814ceb93acb2f41c69d1f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:55:30.253348 31140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6624d5c725814ceb93acb2f41c69d1f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:55:30.253530 31140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6624d5c725814ceb93acb2f41c69d1f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:30.254951 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:55:30.256932 31140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6624d5c725814ceb93acb2f41c69d1f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6624d5c725814ceb93acb2f41c69d1f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32967 } }
I20260928 16:55:30.257694 31140 leader_election.cc:302] T 00000000000000000000000000000000 P 6624d5c725814ceb93acb2f41c69d1f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6624d5c725814ceb93acb2f41c69d1f8; no voters:
I20260928 16:55:30.258522 31140 leader_election.cc:288] T 00000000000000000000000000000000 P 6624d5c725814ceb93acb2f41c69d1f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:55:30.258675 31145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6624d5c725814ceb93acb2f41c69d1f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:55:30.260191 31145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6624d5c725814ceb93acb2f41c69d1f8 [term 1 LEADER]: Becoming Leader. State: Replica: 6624d5c725814ceb93acb2f41c69d1f8, State: Running, Role: LEADER
I20260928 16:55:30.260913 31145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6624d5c725814ceb93acb2f41c69d1f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6624d5c725814ceb93acb2f41c69d1f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32967 } }
I20260928 16:55:30.261413 31140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6624d5c725814ceb93acb2f41c69d1f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:55:30.264429 31150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6624d5c725814ceb93acb2f41c69d1f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6624d5c725814ceb93acb2f41c69d1f8. Latest consensus state: current_term: 1 leader_uuid: "6624d5c725814ceb93acb2f41c69d1f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6624d5c725814ceb93acb2f41c69d1f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32967 } } }
I20260928 16:55:30.264916 31150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6624d5c725814ceb93acb2f41c69d1f8 [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:30.265560 31148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6624d5c725814ceb93acb2f41c69d1f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6624d5c725814ceb93acb2f41c69d1f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6624d5c725814ceb93acb2f41c69d1f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32967 } } }
I20260928 16:55:30.265923 31148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6624d5c725814ceb93acb2f41c69d1f8 [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:30.267648 31156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:55:30.275607 31156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:55:30.287132 31143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:30.262119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.222:47335 (local address 127.0.0.1:32967)
0928 16:55:30.262309 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:30.262320 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:30.268070 (+ 5750us) server_negotiation.cc:407] Connection header received
0928 16:55:30.268278 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:30.268285 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:30.268390 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:30.268496 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:30.270941 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:30.272184 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:30.274189 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:30.274892 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:30.275025 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:30.278793 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:30.278802 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:30.278807 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:30.278859 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:30.282862 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:30.284158 (+ 1296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:30.284170 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:30.284174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:30.284370 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:30.285057 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:30.285062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:30.285065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:30.285568 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:30.285934 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:30.286463 (+ 529us) server_negotiation.cc:299] Negotiation successful
0928 16:55:30.286700 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 16:55:30.294375 31156 catalog_manager.cc:1347] Generated new cluster ID: 312390e6eb624766bfc3dca2b3006613
I20260928 16:55:30.294508 31156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:55:30.313262 31156 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:55:30.314759 31156 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:55:30.332274 31156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6624d5c725814ceb93acb2f41c69d1f8: Generated new TSK 0
I20260928 16:55:30.333323 31156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:55:30.349454 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 31073
I20260928 16:55:30.350984 31135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:55:30.449723 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 31073
W20260928 16:55:30.467248 31058 connection.cc:535] client connection to 127.0.0.1:32967 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32967): Connection reset by peer (error 104)
2026-09-28T16:55:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2116 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:55:30 dist-test-slave-wzcb krb5kdc[31173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:55:30 dist-test-slave-wzcb krb5kdc[31173](info): set up 2 sockets
Sep 28 16:55:30 dist-test-slave-wzcb krb5kdc[31173](info): commencing operation
krb5kdc: starting...
W20260928 16:55:31.502142 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:55:31 dist-test-slave-wzcb krb5kdc[31173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:55:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:55:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:55:31.815433 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35359
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:55:32.193661 31186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:32.194024 31186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:32.194130 31186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:32.201960 31186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:32.202111 31186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:32.202162 31186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:32.202224 31186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:32.202268 31186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:32.213071 31186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35359
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:32.214617 31186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:32.216210 31186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:32.229600 31194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:32.229701 31192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:32.229660 31191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:32.230335 31186 server_base.cc:1034] running on GCE node
I20260928 16:55:32.231393 31186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:32.233191 31186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:32.234454 31186 hybrid_clock.cc:648] HybridClock initialized: now 1790614532234383 us; error 68 us; skew 500 ppm
I20260928 16:55:32.234891 31186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:32 dist-test-slave-wzcb krb5kdc[31173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:32.238523 31186 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:55:32.240378 31186 webserver.cc:460] Webserver started at http://127.0.0.1:45223/ using document root <none> and password file <none>
I20260928 16:55:32.241191 31186 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:32.241310 31186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:32.241626 31186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:32.245391 31186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "75e1912935fe41b1b330f8687007f574"
format_stamp: "Formatted at 2026-09-28 16:55:32 on dist-test-slave-wzcb"
server_key: "14db5f8de3e5827e82c06dc6a9a869b5"
server_key_iv: "eaa6aadc89d805e9b97c2e1921761590"
server_key_version: "encryptionkey@0"
I20260928 16:55:32.246295 31186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "75e1912935fe41b1b330f8687007f574"
format_stamp: "Formatted at 2026-09-28 16:55:32 on dist-test-slave-wzcb"
server_key: "14db5f8de3e5827e82c06dc6a9a869b5"
server_key_iv: "eaa6aadc89d805e9b97c2e1921761590"
server_key_version: "encryptionkey@0"
I20260928 16:55:32.252871 31186 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260928 16:55:32.257299 31201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:32.259097 31186 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 16:55:32.259289 31186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "75e1912935fe41b1b330f8687007f574"
format_stamp: "Formatted at 2026-09-28 16:55:32 on dist-test-slave-wzcb"
server_key: "14db5f8de3e5827e82c06dc6a9a869b5"
server_key_iv: "eaa6aadc89d805e9b97c2e1921761590"
server_key_version: "encryptionkey@0"
I20260928 16:55:32.259572 31186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:32.275816 31186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:32.281250 31186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:32.281615 31186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:32.302069 31186 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35359
I20260928 16:55:32.302093 31252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35359 every 8 connection(s)
I20260928 16:55:32.303761 31186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:55:32.307962 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 31186
I20260928 16:55:32.308171 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:55:32.308295 31253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:32.308681 9588 external_mini_cluster.cc:1428] Setting key 3ef175a7c9cfa854a8ea47ec8382439f
I20260928 16:55:32.324609 31253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75e1912935fe41b1b330f8687007f574: Bootstrap starting.
I20260928 16:55:32.325101 31256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:32.310495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.222:45879 (local address 127.0.0.1:35359)
0928 16:55:32.311429 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:32.311445 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:32.311498 (+ 53us) server_negotiation.cc:407] Connection header received
0928 16:55:32.314224 (+ 2726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:32.314256 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:32.314814 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:32.315248 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:32.315986 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:32.317503 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:32.319655 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:32.320513 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:32.320696 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:32.322202 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:32.322252 (+ 50us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:32.322272 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:32.322934 (+ 662us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:32.323032 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:32.323061 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:32.323201 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:32.323611 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 16:55:32.323855 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":386,"threads_started":1}
I20260928 16:55:32.328866 31253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75e1912935fe41b1b330f8687007f574: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:32.330183 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:55:32.330320 31253 log.cc:824] T 00000000000000000000000000000000 P 75e1912935fe41b1b330f8687007f574: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:32.334012 31253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75e1912935fe41b1b330f8687007f574: No bootstrap required, opened a new log
I20260928 16:55:32.343062 31253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75e1912935fe41b1b330f8687007f574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75e1912935fe41b1b330f8687007f574" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35359 } }
I20260928 16:55:32.343616 31253 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75e1912935fe41b1b330f8687007f574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:32.343729 31253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75e1912935fe41b1b330f8687007f574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75e1912935fe41b1b330f8687007f574, State: Initialized, Role: FOLLOWER
I20260928 16:55:32.344467 31253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75e1912935fe41b1b330f8687007f574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75e1912935fe41b1b330f8687007f574" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35359 } }
I20260928 16:55:32.344751 31253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75e1912935fe41b1b330f8687007f574 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:55:32.344926 31253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75e1912935fe41b1b330f8687007f574 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:55:32.345129 31253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75e1912935fe41b1b330f8687007f574 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:32.347911 31256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:32.340061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.222:58559 (local address 127.0.0.1:35359)
0928 16:55:32.340284 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:32.340291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:32.340604 (+ 313us) server_negotiation.cc:407] Connection header received
0928 16:55:32.340770 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:32.340776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:32.340882 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:32.341021 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:32.341668 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:32.342785 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:32.344677 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:32.345371 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:32.345525 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:32.346959 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:32.346968 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:32.346973 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:32.347003 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:32.347081 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:32.347108 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:32.347228 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:32.347508 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 16:55:32.347572 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 16:55:32.348137 31253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75e1912935fe41b1b330f8687007f574 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75e1912935fe41b1b330f8687007f574" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35359 } }
I20260928 16:55:32.348757 31253 leader_election.cc:302] T 00000000000000000000000000000000 P 75e1912935fe41b1b330f8687007f574 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75e1912935fe41b1b330f8687007f574; no voters:
I20260928 16:55:32.349455 31253 leader_election.cc:288] T 00000000000000000000000000000000 P 75e1912935fe41b1b330f8687007f574 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:55:32.349604 31263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75e1912935fe41b1b330f8687007f574 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:55:32.350726 31263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75e1912935fe41b1b330f8687007f574 [term 1 LEADER]: Becoming Leader. State: Replica: 75e1912935fe41b1b330f8687007f574, State: Running, Role: LEADER
I20260928 16:55:32.351359 31263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75e1912935fe41b1b330f8687007f574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75e1912935fe41b1b330f8687007f574" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35359 } }
I20260928 16:55:32.352020 31253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75e1912935fe41b1b330f8687007f574 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:55:32.355095 31265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75e1912935fe41b1b330f8687007f574 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75e1912935fe41b1b330f8687007f574. Latest consensus state: current_term: 1 leader_uuid: "75e1912935fe41b1b330f8687007f574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75e1912935fe41b1b330f8687007f574" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35359 } } }
I20260928 16:55:32.355623 31265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75e1912935fe41b1b330f8687007f574 [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:32.355067 31264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75e1912935fe41b1b330f8687007f574 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75e1912935fe41b1b330f8687007f574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75e1912935fe41b1b330f8687007f574" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35359 } } }
I20260928 16:55:32.356946 31264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75e1912935fe41b1b330f8687007f574 [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:32.357396 31272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:55:32.363417 31272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:55:32.376874 31272 catalog_manager.cc:1347] Generated new cluster ID: 4ecc05d8eeb54685850a713a71b8091c
I20260928 16:55:32.377025 31272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:55:32.399281 31272 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:55:32.400837 31272 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:55:32.411350 31272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75e1912935fe41b1b330f8687007f574: Generated new TSK 0
I20260928 16:55:32.412437 31272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:55:32.423115 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 31186
W20260928 16:55:32.424861 31248 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.222:45879
W20260928 16:55:32.425794 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 31186: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:55:32.425978 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 31186
W20260928 16:55:32.443485 31171 connection.cc:535] client connection to 127.0.0.1:35359 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35359): Connection reset by peer (error 104)
2026-09-28T16:55:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1975 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:55:32 dist-test-slave-wzcb krb5kdc[31284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:55:32 dist-test-slave-wzcb krb5kdc[31284](info): set up 2 sockets
Sep 28 16:55:32 dist-test-slave-wzcb krb5kdc[31284](info): commencing operation
krb5kdc: starting...
W20260928 16:55:33.490265 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.007s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:55:33 dist-test-slave-wzcb krb5kdc[31284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:55:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:55:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:55:33.803468 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42413
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:55:34.179654 31297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:34.180038 31297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:34.180198 31297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:34.188012 31297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:34.188172 31297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:34.188270 31297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:34.188354 31297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:34.188418 31297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:34.199198 31297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42413
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:34.200793 31297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:34.202350 31297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:34.216106 31305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:34.216158 31302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:34.216193 31303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:34.217104 31297 server_base.cc:1034] running on GCE node
I20260928 16:55:34.218057 31297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:34.219832 31297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:34.221086 31297 hybrid_clock.cc:648] HybridClock initialized: now 1790614534221025 us; error 51 us; skew 500 ppm
I20260928 16:55:34.221519 31297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:34 dist-test-slave-wzcb krb5kdc[31284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:34.230926 31297 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:55:34.232947 31297 webserver.cc:460] Webserver started at http://127.0.0.1:39061/ using document root <none> and password file <none>
I20260928 16:55:34.233806 31297 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:34.233969 31297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:34.234310 31297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:34.237844 31297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "25d3bfa2ff5244c48ba3f219371a6e02"
format_stamp: "Formatted at 2026-09-28 16:55:34 on dist-test-slave-wzcb"
server_key: "bd02eb47b9fcde8ba5ad3d6189e1c732"
server_key_iv: "eda3f366a59e03777d9308d492699e75"
server_key_version: "encryptionkey@0"
I20260928 16:55:34.238750 31297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "25d3bfa2ff5244c48ba3f219371a6e02"
format_stamp: "Formatted at 2026-09-28 16:55:34 on dist-test-slave-wzcb"
server_key: "bd02eb47b9fcde8ba5ad3d6189e1c732"
server_key_iv: "eda3f366a59e03777d9308d492699e75"
server_key_version: "encryptionkey@0"
I20260928 16:55:34.244874 31297 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 16:55:34.249138 31312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:34.250761 31297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260928 16:55:34.250981 31297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "25d3bfa2ff5244c48ba3f219371a6e02"
format_stamp: "Formatted at 2026-09-28 16:55:34 on dist-test-slave-wzcb"
server_key: "bd02eb47b9fcde8ba5ad3d6189e1c732"
server_key_iv: "eda3f366a59e03777d9308d492699e75"
server_key_version: "encryptionkey@0"
I20260928 16:55:34.251206 31297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:34.257776 31297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:34.262493 31297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:34.262821 31297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:34.282501 31297 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42413
I20260928 16:55:34.282524 31363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42413 every 8 connection(s)
I20260928 16:55:34.284185 31297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:55:34.284768 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 31297
I20260928 16:55:34.284978 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:55:34.285570 9588 external_mini_cluster.cc:1428] Setting key 9728c16d93d6f4a18f87174ba3cbed18
I20260928 16:55:34.289601 31364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:34.295588 31367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:34.287750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.222:54463 (local address 127.0.0.1:42413)
0928 16:55:34.288608 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:34.288627 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:34.288685 (+ 58us) server_negotiation.cc:407] Connection header received
0928 16:55:34.291099 (+ 2414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:34.291137 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:34.291688 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:34.292199 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:34.293273 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:34.293284 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:34.293310 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:34.293328 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:34.293440 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:34.293469 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:34.293698 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:34.293850 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 16:55:34.294089 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":333,"threads_started":1}
I20260928 16:55:34.300381 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:55:34.304004 31364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25d3bfa2ff5244c48ba3f219371a6e02: Bootstrap starting.
I20260928 16:55:34.307607 31367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:34.305074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.222:44463 (local address 127.0.0.1:42413)
0928 16:55:34.305268 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:34.305274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:34.305542 (+ 268us) server_negotiation.cc:407] Connection header received
0928 16:55:34.305712 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:34.305718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:34.305794 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:34.305930 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:34.306868 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:34.306874 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:34.306878 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:34.306899 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:34.306964 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:34.306987 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:34.307115 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:34.307187 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 16:55:34.307257 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:55:34.308220 31364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25d3bfa2ff5244c48ba3f219371a6e02: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:34.309655 31364 log.cc:824] T 00000000000000000000000000000000 P 25d3bfa2ff5244c48ba3f219371a6e02: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:34.312934 31364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25d3bfa2ff5244c48ba3f219371a6e02: No bootstrap required, opened a new log
I20260928 16:55:34.321182 31364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25d3bfa2ff5244c48ba3f219371a6e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d3bfa2ff5244c48ba3f219371a6e02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42413 } }
I20260928 16:55:34.321657 31364 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25d3bfa2ff5244c48ba3f219371a6e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:34.321766 31364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25d3bfa2ff5244c48ba3f219371a6e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25d3bfa2ff5244c48ba3f219371a6e02, State: Initialized, Role: FOLLOWER
I20260928 16:55:34.322499 31364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25d3bfa2ff5244c48ba3f219371a6e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d3bfa2ff5244c48ba3f219371a6e02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42413 } }
I20260928 16:55:34.322741 31364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25d3bfa2ff5244c48ba3f219371a6e02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:55:34.322853 31364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25d3bfa2ff5244c48ba3f219371a6e02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:55:34.323000 31364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25d3bfa2ff5244c48ba3f219371a6e02 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:34.325348 31364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25d3bfa2ff5244c48ba3f219371a6e02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d3bfa2ff5244c48ba3f219371a6e02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42413 } }
I20260928 16:55:34.325889 31364 leader_election.cc:302] T 00000000000000000000000000000000 P 25d3bfa2ff5244c48ba3f219371a6e02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25d3bfa2ff5244c48ba3f219371a6e02; no voters:
I20260928 16:55:34.326527 31364 leader_election.cc:288] T 00000000000000000000000000000000 P 25d3bfa2ff5244c48ba3f219371a6e02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:55:34.326683 31374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25d3bfa2ff5244c48ba3f219371a6e02 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:55:34.327428 31374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25d3bfa2ff5244c48ba3f219371a6e02 [term 1 LEADER]: Becoming Leader. State: Replica: 25d3bfa2ff5244c48ba3f219371a6e02, State: Running, Role: LEADER
I20260928 16:55:34.328025 31374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25d3bfa2ff5244c48ba3f219371a6e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d3bfa2ff5244c48ba3f219371a6e02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42413 } }
I20260928 16:55:34.329128 31364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25d3bfa2ff5244c48ba3f219371a6e02 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:55:34.331159 31376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25d3bfa2ff5244c48ba3f219371a6e02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25d3bfa2ff5244c48ba3f219371a6e02. Latest consensus state: current_term: 1 leader_uuid: "25d3bfa2ff5244c48ba3f219371a6e02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d3bfa2ff5244c48ba3f219371a6e02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42413 } } }
I20260928 16:55:34.331526 31375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25d3bfa2ff5244c48ba3f219371a6e02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25d3bfa2ff5244c48ba3f219371a6e02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d3bfa2ff5244c48ba3f219371a6e02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42413 } } }
I20260928 16:55:34.331729 31376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25d3bfa2ff5244c48ba3f219371a6e02 [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:34.331782 31375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25d3bfa2ff5244c48ba3f219371a6e02 [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:34.333941 31382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:55:34.339943 31382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:55:34.350656 31382 catalog_manager.cc:1347] Generated new cluster ID: b2b577085dc34cf9b6ae8aeec0841897
I20260928 16:55:34.350800 31382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:55:34.365585 31382 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:55:34.366099 31382 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:55:34.372352 31382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25d3bfa2ff5244c48ba3f219371a6e02: Generated new TSK 0
I20260928 16:55:34.373314 31382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:55:34.387826 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 31297
W20260928 16:55:34.389315 31359 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.222:54463
W20260928 16:55:34.390219 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 31297: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:55:34.390376 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 31297
2026-09-28T16:55:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1963 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:55:34 dist-test-slave-wzcb krb5kdc[31395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:55:34 dist-test-slave-wzcb krb5kdc[31395](info): set up 2 sockets
Sep 28 16:55:34 dist-test-slave-wzcb krb5kdc[31395](info): commencing operation
krb5kdc: starting...
W20260928 16:55:35.438206 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.009s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:55:35 dist-test-slave-wzcb krb5kdc[31395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:55:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:55:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:55:35.750618 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:35985
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:43901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:35985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:55:36.129851 31408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:36.130198 31408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:36.130352 31408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:36.138110 31408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:36.138252 31408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:36.138335 31408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:36.138420 31408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:36.138486 31408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:36.149197 31408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:43901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:35985
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:35985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:36.150768 31408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:36.152488 31408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:36.165714 31413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:36.165777 31414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:36.165793 31416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:36.166769 31408 server_base.cc:1034] running on GCE node
I20260928 16:55:36.167770 31408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:36.169445 31408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:36.170665 31408 hybrid_clock.cc:648] HybridClock initialized: now 1790614536170594 us; error 61 us; skew 500 ppm
I20260928 16:55:36.171069 31408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:36 dist-test-slave-wzcb krb5kdc[31395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614536, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:36.174867 31408 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:55:36.176750 31408 webserver.cc:460] Webserver started at http://127.9.93.62:41495/ using document root <none> and password file <none>
I20260928 16:55:36.177610 31408 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:36.177755 31408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:36.178100 31408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:36.181921 31408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "7bc0ec31e3bf4fd9aafe3f4408e27419"
format_stamp: "Formatted at 2026-09-28 16:55:36 on dist-test-slave-wzcb"
server_key: "a85bdb69a34de40076252a3efb4b3a14"
server_key_iv: "e497f591f1396649dc86879d4fc016df"
server_key_version: "encryptionkey@0"
I20260928 16:55:36.182871 31408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "7bc0ec31e3bf4fd9aafe3f4408e27419"
format_stamp: "Formatted at 2026-09-28 16:55:36 on dist-test-slave-wzcb"
server_key: "a85bdb69a34de40076252a3efb4b3a14"
server_key_iv: "e497f591f1396649dc86879d4fc016df"
server_key_version: "encryptionkey@0"
I20260928 16:55:36.189574 31408 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260928 16:55:36.194187 31423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:36.195963 31408 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:55:36.196184 31408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "7bc0ec31e3bf4fd9aafe3f4408e27419"
format_stamp: "Formatted at 2026-09-28 16:55:36 on dist-test-slave-wzcb"
server_key: "a85bdb69a34de40076252a3efb4b3a14"
server_key_iv: "e497f591f1396649dc86879d4fc016df"
server_key_version: "encryptionkey@0"
I20260928 16:55:36.196441 31408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:36.212159 31408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:36.217226 31408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:36.217566 31408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:36.237228 31408 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:35985
I20260928 16:55:36.237252 31474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:35985 every 8 connection(s)
I20260928 16:55:36.238866 31408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:55:36.243261 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 31408
I20260928 16:55:36.243353 31475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:36.243515 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:55:36.244021 9588 external_mini_cluster.cc:1428] Setting key 8271f1438967ce2a5c0f0014d161103e
I20260928 16:55:36.259739 31475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bc0ec31e3bf4fd9aafe3f4408e27419: Bootstrap starting.
Sep 28 16:55:36 dist-test-slave-wzcb krb5kdc[31395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:55:36.263942 31475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bc0ec31e3bf4fd9aafe3f4408e27419: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:36.265280 31475 log.cc:824] T 00000000000000000000000000000000 P 7bc0ec31e3bf4fd9aafe3f4408e27419: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:36.269098 31475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bc0ec31e3bf4fd9aafe3f4408e27419: No bootstrap required, opened a new log
I20260928 16:55:36.270629 31478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:36.246142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37145 (local address 127.9.93.62:35985)
0928 16:55:36.247103 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:36.247117 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:36.247171 (+ 54us) server_negotiation.cc:407] Connection header received
0928 16:55:36.248956 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:36.248984 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:36.249518 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:36.249954 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:36.252057 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:36.253516 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:36.255467 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:36.256403 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:36.256566 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:36.262080 (+ 5514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:36.262091 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:36.262108 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:36.262163 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:36.265622 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:36.266743 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:36.266752 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:36.266761 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:36.266976 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:36.267588 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:36.267593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:36.267595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:36.268218 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:36.268505 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:36.269083 (+ 578us) server_negotiation.cc:299] Negotiation successful
0928 16:55:36.269467 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":431,"threads_started":1}
I20260928 16:55:36.275233 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:55:36.277925 31475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bc0ec31e3bf4fd9aafe3f4408e27419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc0ec31e3bf4fd9aafe3f4408e27419" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 35985 } }
I20260928 16:55:36.278453 31475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bc0ec31e3bf4fd9aafe3f4408e27419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:36.278580 31475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bc0ec31e3bf4fd9aafe3f4408e27419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bc0ec31e3bf4fd9aafe3f4408e27419, State: Initialized, Role: FOLLOWER
I20260928 16:55:36.279348 31475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bc0ec31e3bf4fd9aafe3f4408e27419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc0ec31e3bf4fd9aafe3f4408e27419" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 35985 } }
I20260928 16:55:36.279749 31475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bc0ec31e3bf4fd9aafe3f4408e27419 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:55:36.279955 31475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bc0ec31e3bf4fd9aafe3f4408e27419 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:55:36.280221 31475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bc0ec31e3bf4fd9aafe3f4408e27419 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:36.284020 31475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bc0ec31e3bf4fd9aafe3f4408e27419 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc0ec31e3bf4fd9aafe3f4408e27419" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 35985 } }
I20260928 16:55:36.284674 31475 leader_election.cc:302] T 00000000000000000000000000000000 P 7bc0ec31e3bf4fd9aafe3f4408e27419 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bc0ec31e3bf4fd9aafe3f4408e27419; no voters:
I20260928 16:55:36.285334 31475 leader_election.cc:288] T 00000000000000000000000000000000 P 7bc0ec31e3bf4fd9aafe3f4408e27419 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:55:36.285513 31485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bc0ec31e3bf4fd9aafe3f4408e27419 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:55:36.286688 31485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bc0ec31e3bf4fd9aafe3f4408e27419 [term 1 LEADER]: Becoming Leader. State: Replica: 7bc0ec31e3bf4fd9aafe3f4408e27419, State: Running, Role: LEADER
I20260928 16:55:36.287441 31485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bc0ec31e3bf4fd9aafe3f4408e27419 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc0ec31e3bf4fd9aafe3f4408e27419" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 35985 } }
I20260928 16:55:36.288138 31475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bc0ec31e3bf4fd9aafe3f4408e27419 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:55:36.290295 31486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc0ec31e3bf4fd9aafe3f4408e27419 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bc0ec31e3bf4fd9aafe3f4408e27419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc0ec31e3bf4fd9aafe3f4408e27419" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 35985 } } }
I20260928 16:55:36.290751 31486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc0ec31e3bf4fd9aafe3f4408e27419 [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:36.291357 31487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc0ec31e3bf4fd9aafe3f4408e27419 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bc0ec31e3bf4fd9aafe3f4408e27419. Latest consensus state: current_term: 1 leader_uuid: "7bc0ec31e3bf4fd9aafe3f4408e27419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc0ec31e3bf4fd9aafe3f4408e27419" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 35985 } } }
I20260928 16:55:36.291675 31487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc0ec31e3bf4fd9aafe3f4408e27419 [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:36.292059 31490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:55:36.304708 31478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:36.279958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40195 (local address 127.9.93.62:35985)
0928 16:55:36.280143 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:36.280148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:36.284013 (+ 3865us) server_negotiation.cc:407] Connection header received
0928 16:55:36.284215 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:36.284220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:36.284314 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:36.284420 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:36.286458 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:36.287767 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:36.289681 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:36.290420 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:36.290591 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:36.294351 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:36.294360 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:36.294367 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:36.294422 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:36.298665 (+ 4243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:36.300198 (+ 1533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:36.300206 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:36.300210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:36.300419 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:36.301106 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:36.301111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:36.301115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:36.301810 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:36.302159 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:36.303672 (+ 1513us) server_negotiation.cc:299] Negotiation successful
0928 16:55:36.303928 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:55:36.310214 31490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:55:36.323454 31490 catalog_manager.cc:1347] Generated new cluster ID: 7563c626830a427d980b0c8aa67208b8
I20260928 16:55:36.323670 31490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:55:36.354401 31490 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:55:36.356127 31490 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:55:36.368763 31490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bc0ec31e3bf4fd9aafe3f4408e27419: Generated new TSK 0
I20260928 16:55:36.369766 31490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:55:36.385995 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 31408
I20260928 16:55:36.387440 31470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:55:36.485668 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 31408
W20260928 16:55:36.503052 31393 connection.cc:535] client connection to 127.9.93.62:35985 recv error: Network error: failed to read from TLS socket (remote: 127.9.93.62:35985): Connection reset by peer (error 104)
2026-09-28T16:55:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2095 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:55:36 dist-test-slave-wzcb krb5kdc[31507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:55:36 dist-test-slave-wzcb krb5kdc[31507](info): set up 2 sockets
Sep 28 16:55:36 dist-test-slave-wzcb krb5kdc[31507](info): commencing operation
krb5kdc: starting...
W20260928 16:55:37.542223 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:55:37 dist-test-slave-wzcb krb5kdc[31507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:55:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:55:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:55:37.859889 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:41961
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:36871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:41961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:55:38.238981 31520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:38.239425 31520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:38.239595 31520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:38.247430 31520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:38.247589 31520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:38.247675 31520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:38.247764 31520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:38.247830 31520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:38.258755 31520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:36871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:41961
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:41961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:38.260344 31520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:38.261894 31520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:38.275365 31526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:38.275396 31525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:38.275441 31528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:38.276427 31520 server_base.cc:1034] running on GCE node
I20260928 16:55:38.277340 31520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:38.279079 31520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:38.280330 31520 hybrid_clock.cc:648] HybridClock initialized: now 1790614538280239 us; error 85 us; skew 500 ppm
I20260928 16:55:38.280757 31520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:38 dist-test-slave-wzcb krb5kdc[31507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614538, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:38.284237 31520 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:55:38.286070 31520 webserver.cc:460] Webserver started at http://127.9.93.62:45729/ using document root <none> and password file <none>
I20260928 16:55:38.286906 31520 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:38.287062 31520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:38.287464 31520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:38.290952 31520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "516a33eccc2743d4a29e1771b15257c5"
format_stamp: "Formatted at 2026-09-28 16:55:38 on dist-test-slave-wzcb"
server_key: "ce689dfeb165ce8bf977009592256fa5"
server_key_iv: "c6a52a11083d75909214fdeefc5dec82"
server_key_version: "encryptionkey@0"
I20260928 16:55:38.291868 31520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "516a33eccc2743d4a29e1771b15257c5"
format_stamp: "Formatted at 2026-09-28 16:55:38 on dist-test-slave-wzcb"
server_key: "ce689dfeb165ce8bf977009592256fa5"
server_key_iv: "c6a52a11083d75909214fdeefc5dec82"
server_key_version: "encryptionkey@0"
I20260928 16:55:38.298205 31520 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:55:38.302454 31535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:38.304265 31520 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:55:38.304483 31520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "516a33eccc2743d4a29e1771b15257c5"
format_stamp: "Formatted at 2026-09-28 16:55:38 on dist-test-slave-wzcb"
server_key: "ce689dfeb165ce8bf977009592256fa5"
server_key_iv: "c6a52a11083d75909214fdeefc5dec82"
server_key_version: "encryptionkey@0"
I20260928 16:55:38.304720 31520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:38.324961 31520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:38.330519 31520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:38.330880 31520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:38.352572 31520 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:41961
I20260928 16:55:38.352648 31586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:41961 every 8 connection(s)
I20260928 16:55:38.354197 31520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:55:38.358657 31587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:38.362326 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 31520
I20260928 16:55:38.362524 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:55:38.363003 9588 external_mini_cluster.cc:1428] Setting key e442b7d49b4fe4a1d35d2abfb80f458f
I20260928 16:55:38.373374 31587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 516a33eccc2743d4a29e1771b15257c5: Bootstrap starting.
I20260928 16:55:38.377709 31587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 516a33eccc2743d4a29e1771b15257c5: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:38.379124 31587 log.cc:824] T 00000000000000000000000000000000 P 516a33eccc2743d4a29e1771b15257c5: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:38.379711 31590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:38.365370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38941 (local address 127.9.93.62:41961)
0928 16:55:38.366357 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:38.366372 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:38.366427 (+ 55us) server_negotiation.cc:407] Connection header received
0928 16:55:38.368857 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:38.368890 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:38.369504 (+ 614us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:38.369969 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:38.370687 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:38.372347 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:38.374309 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:38.375157 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:38.375484 (+ 327us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:38.376935 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:38.376944 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:38.376962 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:38.377570 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:38.377671 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:38.377708 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:38.377854 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:38.378188 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 16:55:38.378453 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":467,"threads_started":1}
I20260928 16:55:38.383035 31587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 516a33eccc2743d4a29e1771b15257c5: No bootstrap required, opened a new log
I20260928 16:55:38.384708 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:55:38.394387 31587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 516a33eccc2743d4a29e1771b15257c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516a33eccc2743d4a29e1771b15257c5" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 41961 } }
I20260928 16:55:38.395004 31587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 516a33eccc2743d4a29e1771b15257c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:38.395139 31587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 516a33eccc2743d4a29e1771b15257c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516a33eccc2743d4a29e1771b15257c5, State: Initialized, Role: FOLLOWER
I20260928 16:55:38.395900 31587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 516a33eccc2743d4a29e1771b15257c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516a33eccc2743d4a29e1771b15257c5" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 41961 } }
I20260928 16:55:38.396221 31587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 516a33eccc2743d4a29e1771b15257c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:55:38.396418 31587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 516a33eccc2743d4a29e1771b15257c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:55:38.396621 31587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 516a33eccc2743d4a29e1771b15257c5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:38.398412 31590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:38.390639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50283 (local address 127.9.93.62:41961)
0928 16:55:38.390877 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:38.390885 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:38.391421 (+ 536us) server_negotiation.cc:407] Connection header received
0928 16:55:38.391638 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:38.391643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:38.391723 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:38.391824 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:38.392495 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:38.393339 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:38.395283 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:38.395913 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:38.396017 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:38.397454 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:38.397463 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:38.397467 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:38.397491 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:38.397570 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:38.397594 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:38.397733 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:38.397984 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 16:55:38.398053 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 16:55:38.399782 31587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 516a33eccc2743d4a29e1771b15257c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516a33eccc2743d4a29e1771b15257c5" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 41961 } }
I20260928 16:55:38.400418 31587 leader_election.cc:302] T 00000000000000000000000000000000 P 516a33eccc2743d4a29e1771b15257c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 516a33eccc2743d4a29e1771b15257c5; no voters:
I20260928 16:55:38.401038 31587 leader_election.cc:288] T 00000000000000000000000000000000 P 516a33eccc2743d4a29e1771b15257c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:55:38.401176 31597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 516a33eccc2743d4a29e1771b15257c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:55:38.401923 31597 raft_consensus.cc:686] T 00000000000000000000000000000000 P 516a33eccc2743d4a29e1771b15257c5 [term 1 LEADER]: Becoming Leader. State: Replica: 516a33eccc2743d4a29e1771b15257c5, State: Running, Role: LEADER
I20260928 16:55:38.402554 31597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 516a33eccc2743d4a29e1771b15257c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516a33eccc2743d4a29e1771b15257c5" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 41961 } }
I20260928 16:55:38.403229 31587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 516a33eccc2743d4a29e1771b15257c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:55:38.405818 31598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 516a33eccc2743d4a29e1771b15257c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "516a33eccc2743d4a29e1771b15257c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516a33eccc2743d4a29e1771b15257c5" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 41961 } } }
I20260928 16:55:38.406267 31598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 516a33eccc2743d4a29e1771b15257c5 [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:38.407087 31599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 516a33eccc2743d4a29e1771b15257c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 516a33eccc2743d4a29e1771b15257c5. Latest consensus state: current_term: 1 leader_uuid: "516a33eccc2743d4a29e1771b15257c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516a33eccc2743d4a29e1771b15257c5" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 41961 } } }
I20260928 16:55:38.407569 31599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 516a33eccc2743d4a29e1771b15257c5 [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:38.407953 31602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:55:38.414273 31602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:55:38.426792 31602 catalog_manager.cc:1347] Generated new cluster ID: f866ff02f74f44829e8a5d0008640d5f
I20260928 16:55:38.426949 31602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:55:38.443512 31602 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:55:38.445106 31602 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:55:38.466710 31602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 516a33eccc2743d4a29e1771b15257c5: Generated new TSK 0
I20260928 16:55:38.468057 31602 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:55:38.482015 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 31520
W20260928 16:55:38.484586 31582 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38941
W20260928 16:55:38.485518 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 31520: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:55:38.485704 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 31520
W20260928 16:55:38.503098 31505 connection.cc:535] client connection to 127.9.93.62:41961 recv error: Network error: failed to read from TLS socket (remote: 127.9.93.62:41961): Connection reset by peer (error 104)
2026-09-28T16:55:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1999 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:55:38 dist-test-slave-wzcb krb5kdc[31618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:55:38 dist-test-slave-wzcb krb5kdc[31618](info): set up 2 sockets
Sep 28 16:55:38 dist-test-slave-wzcb krb5kdc[31618](info): commencing operation
krb5kdc: starting...
W20260928 16:55:39.542346 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:55:39 dist-test-slave-wzcb krb5kdc[31618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:55:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:55:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:55:39.856001 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:43117
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:40343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:43117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:55:40.249696 31631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:40.250046 31631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:40.250145 31631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:40.258085 31631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:40.258226 31631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:40.258277 31631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:40.258342 31631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:40.258395 31631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:40.269272 31631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:40343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:43117
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:43117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:40.270795 31631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:40.272393 31631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:40.285852 31636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:40.285876 31639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:40.285768 31637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:40.287345 31631 server_base.cc:1034] running on GCE node
I20260928 16:55:40.288313 31631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:40.290055 31631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:40.291285 31631 hybrid_clock.cc:648] HybridClock initialized: now 1790614540291231 us; error 58 us; skew 500 ppm
I20260928 16:55:40.291718 31631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:40 dist-test-slave-wzcb krb5kdc[31618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614540, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:40.295238 31631 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:55:40.297113 31631 webserver.cc:460] Webserver started at http://127.9.93.62:44675/ using document root <none> and password file <none>
I20260928 16:55:40.297964 31631 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:40.298102 31631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:40.298448 31631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:40.302052 31631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "b998841d27224273a62a40a0d247f377"
format_stamp: "Formatted at 2026-09-28 16:55:40 on dist-test-slave-wzcb"
server_key: "bf9474201140b199ddcd1b1aeddbf549"
server_key_iv: "ab9d7ddf9c2093056c77ffce991b6059"
server_key_version: "encryptionkey@0"
I20260928 16:55:40.302990 31631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "b998841d27224273a62a40a0d247f377"
format_stamp: "Formatted at 2026-09-28 16:55:40 on dist-test-slave-wzcb"
server_key: "bf9474201140b199ddcd1b1aeddbf549"
server_key_iv: "ab9d7ddf9c2093056c77ffce991b6059"
server_key_version: "encryptionkey@0"
I20260928 16:55:40.309422 31631 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 16:55:40.313864 31646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:40.315656 31631 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:55:40.315872 31631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "b998841d27224273a62a40a0d247f377"
format_stamp: "Formatted at 2026-09-28 16:55:40 on dist-test-slave-wzcb"
server_key: "bf9474201140b199ddcd1b1aeddbf549"
server_key_iv: "ab9d7ddf9c2093056c77ffce991b6059"
server_key_version: "encryptionkey@0"
I20260928 16:55:40.316125 31631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:40.322773 31631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:40.327854 31631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:40.328202 31631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:40.348258 31631 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:43117
I20260928 16:55:40.348289 31697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:43117 every 8 connection(s)
I20260928 16:55:40.349875 31631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:55:40.354614 31698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:40.359205 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 31631
I20260928 16:55:40.359439 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:55:40.359930 9588 external_mini_cluster.cc:1428] Setting key 95be5e0a3b6a9bb3f7e73130c7f1df63
I20260928 16:55:40.369328 31701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:40.362021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41613 (local address 127.9.93.62:43117)
0928 16:55:40.362926 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:40.362942 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:40.362991 (+ 49us) server_negotiation.cc:407] Connection header received
0928 16:55:40.364844 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:40.364879 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:40.365375 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:40.365822 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:40.367361 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:40.367372 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:40.367396 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:40.367418 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:40.367499 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:40.367530 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:40.367756 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:40.367873 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 16:55:40.368105 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":397,"threads_started":1}
I20260928 16:55:40.369730 31698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b998841d27224273a62a40a0d247f377: Bootstrap starting.
I20260928 16:55:40.373536 31698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b998841d27224273a62a40a0d247f377: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:40.373780 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:55:40.375041 31698 log.cc:824] T 00000000000000000000000000000000 P b998841d27224273a62a40a0d247f377: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:40.379240 31698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b998841d27224273a62a40a0d247f377: No bootstrap required, opened a new log
I20260928 16:55:40.382009 31701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:40.379209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48035 (local address 127.9.93.62:43117)
0928 16:55:40.379449 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:40.379455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:40.379996 (+ 541us) server_negotiation.cc:407] Connection header received
0928 16:55:40.380155 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:40.380160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:40.380237 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:40.380358 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:40.381287 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:40.381296 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:40.381302 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:40.381324 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:40.381387 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:40.381413 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:40.381547 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:40.381616 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 16:55:40.381683 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 16:55:40.387964 31698 raft_consensus.cc:348] T 00000000000000000000000000000000 P b998841d27224273a62a40a0d247f377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b998841d27224273a62a40a0d247f377" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43117 } }
I20260928 16:55:40.388429 31698 raft_consensus.cc:374] T 00000000000000000000000000000000 P b998841d27224273a62a40a0d247f377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:40.388540 31698 raft_consensus.cc:729] T 00000000000000000000000000000000 P b998841d27224273a62a40a0d247f377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b998841d27224273a62a40a0d247f377, State: Initialized, Role: FOLLOWER
I20260928 16:55:40.389189 31698 consensus_queue.cc:260] T 00000000000000000000000000000000 P b998841d27224273a62a40a0d247f377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b998841d27224273a62a40a0d247f377" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43117 } }
I20260928 16:55:40.389458 31698 raft_consensus.cc:388] T 00000000000000000000000000000000 P b998841d27224273a62a40a0d247f377 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:55:40.389607 31698 raft_consensus.cc:482] T 00000000000000000000000000000000 P b998841d27224273a62a40a0d247f377 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:55:40.389792 31698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b998841d27224273a62a40a0d247f377 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:40.392338 31698 raft_consensus.cc:504] T 00000000000000000000000000000000 P b998841d27224273a62a40a0d247f377 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b998841d27224273a62a40a0d247f377" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43117 } }
I20260928 16:55:40.392997 31698 leader_election.cc:302] T 00000000000000000000000000000000 P b998841d27224273a62a40a0d247f377 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b998841d27224273a62a40a0d247f377; no voters:
I20260928 16:55:40.393994 31698 leader_election.cc:288] T 00000000000000000000000000000000 P b998841d27224273a62a40a0d247f377 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:55:40.394104 31708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b998841d27224273a62a40a0d247f377 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:55:40.394767 31708 raft_consensus.cc:686] T 00000000000000000000000000000000 P b998841d27224273a62a40a0d247f377 [term 1 LEADER]: Becoming Leader. State: Replica: b998841d27224273a62a40a0d247f377, State: Running, Role: LEADER
I20260928 16:55:40.395458 31708 consensus_queue.cc:237] T 00000000000000000000000000000000 P b998841d27224273a62a40a0d247f377 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b998841d27224273a62a40a0d247f377" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43117 } }
I20260928 16:55:40.396426 31698 sys_catalog.cc:564] T 00000000000000000000000000000000 P b998841d27224273a62a40a0d247f377 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:55:40.398263 31710 sys_catalog.cc:455] T 00000000000000000000000000000000 P b998841d27224273a62a40a0d247f377 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b998841d27224273a62a40a0d247f377. Latest consensus state: current_term: 1 leader_uuid: "b998841d27224273a62a40a0d247f377" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b998841d27224273a62a40a0d247f377" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43117 } } }
I20260928 16:55:40.398864 31710 sys_catalog.cc:458] T 00000000000000000000000000000000 P b998841d27224273a62a40a0d247f377 [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:40.398730 31709 sys_catalog.cc:455] T 00000000000000000000000000000000 P b998841d27224273a62a40a0d247f377 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b998841d27224273a62a40a0d247f377" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b998841d27224273a62a40a0d247f377" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 43117 } } }
I20260928 16:55:40.399366 31709 sys_catalog.cc:458] T 00000000000000000000000000000000 P b998841d27224273a62a40a0d247f377 [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:40.407891 31714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:55:40.413885 31714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:55:40.424047 31714 catalog_manager.cc:1347] Generated new cluster ID: d9aca1ba48ce46a48c78c97138a69942
I20260928 16:55:40.424191 31714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:55:40.462222 31714 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:55:40.462781 31714 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:55:40.478062 31714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b998841d27224273a62a40a0d247f377: Generated new TSK 0
I20260928 16:55:40.479216 31714 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:55:40.497431 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 31631
W20260928 16:55:40.498840 31693 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41613
W20260928 16:55:40.499680 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 31631: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:55:40.499836 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 31631
2026-09-28T16:55:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2014 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:55:40 dist-test-slave-wzcb krb5kdc[31729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:55:40 dist-test-slave-wzcb krb5kdc[31729](info): set up 2 sockets
Sep 28 16:55:40 dist-test-slave-wzcb krb5kdc[31729](info): commencing operation
krb5kdc: starting...
W20260928 16:55:41.546131 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:55:41 dist-test-slave-wzcb krb5kdc[31729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:55:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:55:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:55:41.864166 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:40843
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:40827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:40843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:55:42.243039 31742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:42.243469 31742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:42.243640 31742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:42.251539 31742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:42.251686 31742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:42.251766 31742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:42.251847 31742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:42.251907 31742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:42.262758 31742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:40827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:40843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:40843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:42.264384 31742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:42.265926 31742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:42.278905 31747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:42.278952 31750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:42.278875 31748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:42.279913 31742 server_base.cc:1034] running on GCE node
I20260928 16:55:42.280865 31742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:42.282606 31742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:42.283865 31742 hybrid_clock.cc:648] HybridClock initialized: now 1790614542283790 us; error 66 us; skew 500 ppm
I20260928 16:55:42.284339 31742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:42 dist-test-slave-wzcb krb5kdc[31729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614542, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:42.287858 31742 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:55:42.289746 31742 webserver.cc:460] Webserver started at http://127.9.93.62:41827/ using document root <none> and password file <none>
I20260928 16:55:42.290606 31742 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:42.290753 31742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:42.291126 31742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:42.295070 31742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "dcfec095d6df49c8ab4227bea915373e"
format_stamp: "Formatted at 2026-09-28 16:55:42 on dist-test-slave-wzcb"
server_key: "b4d3bae6d1ab7ffa8eb1589b40a1727d"
server_key_iv: "0b62ad3b4ce5e3ec64c086670fe6182a"
server_key_version: "encryptionkey@0"
I20260928 16:55:42.296154 31742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "dcfec095d6df49c8ab4227bea915373e"
format_stamp: "Formatted at 2026-09-28 16:55:42 on dist-test-slave-wzcb"
server_key: "b4d3bae6d1ab7ffa8eb1589b40a1727d"
server_key_iv: "0b62ad3b4ce5e3ec64c086670fe6182a"
server_key_version: "encryptionkey@0"
I20260928 16:55:42.302865 31742 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 16:55:42.307336 31757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:42.309060 31742 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:55:42.309293 31742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "dcfec095d6df49c8ab4227bea915373e"
format_stamp: "Formatted at 2026-09-28 16:55:42 on dist-test-slave-wzcb"
server_key: "b4d3bae6d1ab7ffa8eb1589b40a1727d"
server_key_iv: "0b62ad3b4ce5e3ec64c086670fe6182a"
server_key_version: "encryptionkey@0"
I20260928 16:55:42.309525 31742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:42.336653 31742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:42.341698 31742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:42.342027 31742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:42.362280 31742 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:40843
I20260928 16:55:42.362291 31808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:40843 every 8 connection(s)
I20260928 16:55:42.363917 31742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:55:42.366165 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 31742
I20260928 16:55:42.366403 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:55:42.366847 9588 external_mini_cluster.cc:1428] Setting key 9ef990ccfb8155d0a49b72b16a8b5857
I20260928 16:55:42.368968 31809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 16:55:42 dist-test-slave-wzcb krb5kdc[31729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:55:42.383993 31809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcfec095d6df49c8ab4227bea915373e: Bootstrap starting.
I20260928 16:55:42.388052 31809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcfec095d6df49c8ab4227bea915373e: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:42.389593 31809 log.cc:824] T 00000000000000000000000000000000 P dcfec095d6df49c8ab4227bea915373e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:42.393198 31812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:42.368785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38115 (local address 127.9.93.62:40843)
0928 16:55:42.369654 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:42.369669 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:42.369725 (+ 56us) server_negotiation.cc:407] Connection header received
0928 16:55:42.371599 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:42.371624 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:42.372128 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:42.372535 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:42.374495 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:42.376110 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:42.378007 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:42.378884 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:42.379074 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:42.384574 (+ 5500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:42.384584 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:42.384609 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:42.384667 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:42.388061 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:42.389154 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:42.389163 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:42.389171 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:42.389442 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:42.389952 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:42.389956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:42.389959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:42.390864 (+ 905us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:42.391224 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:42.391590 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 16:55:42.392023 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":384,"threads_started":1}
I20260928 16:55:42.393472 31809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcfec095d6df49c8ab4227bea915373e: No bootstrap required, opened a new log
I20260928 16:55:42.397542 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:55:42.403896 31809 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcfec095d6df49c8ab4227bea915373e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcfec095d6df49c8ab4227bea915373e" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 40843 } }
I20260928 16:55:42.404482 31809 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcfec095d6df49c8ab4227bea915373e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:42.404598 31809 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcfec095d6df49c8ab4227bea915373e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfec095d6df49c8ab4227bea915373e, State: Initialized, Role: FOLLOWER
I20260928 16:55:42.405247 31809 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcfec095d6df49c8ab4227bea915373e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcfec095d6df49c8ab4227bea915373e" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 40843 } }
I20260928 16:55:42.405493 31809 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcfec095d6df49c8ab4227bea915373e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:55:42.405654 31809 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcfec095d6df49c8ab4227bea915373e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:55:42.405827 31809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcfec095d6df49c8ab4227bea915373e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:42.408854 31809 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcfec095d6df49c8ab4227bea915373e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcfec095d6df49c8ab4227bea915373e" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 40843 } }
I20260928 16:55:42.409649 31809 leader_election.cc:302] T 00000000000000000000000000000000 P dcfec095d6df49c8ab4227bea915373e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcfec095d6df49c8ab4227bea915373e; no voters:
I20260928 16:55:42.410529 31809 leader_election.cc:288] T 00000000000000000000000000000000 P dcfec095d6df49c8ab4227bea915373e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:55:42.410710 31819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcfec095d6df49c8ab4227bea915373e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:55:42.411423 31819 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcfec095d6df49c8ab4227bea915373e [term 1 LEADER]: Becoming Leader. State: Replica: dcfec095d6df49c8ab4227bea915373e, State: Running, Role: LEADER
I20260928 16:55:42.412201 31819 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcfec095d6df49c8ab4227bea915373e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcfec095d6df49c8ab4227bea915373e" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 40843 } }
I20260928 16:55:42.415021 31809 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcfec095d6df49c8ab4227bea915373e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:55:42.416044 31820 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcfec095d6df49c8ab4227bea915373e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcfec095d6df49c8ab4227bea915373e. Latest consensus state: current_term: 1 leader_uuid: "dcfec095d6df49c8ab4227bea915373e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcfec095d6df49c8ab4227bea915373e" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 40843 } } }
I20260928 16:55:42.416595 31820 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcfec095d6df49c8ab4227bea915373e [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:42.416941 31821 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcfec095d6df49c8ab4227bea915373e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcfec095d6df49c8ab4227bea915373e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcfec095d6df49c8ab4227bea915373e" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 40843 } } }
I20260928 16:55:42.417327 31821 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcfec095d6df49c8ab4227bea915373e [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:42.419297 31823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:55:42.421182 31812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:42.403342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42015 (local address 127.9.93.62:40843)
0928 16:55:42.403547 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:42.403554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:42.403993 (+ 439us) server_negotiation.cc:407] Connection header received
0928 16:55:42.404141 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:42.404146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:42.404235 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:42.404353 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:42.406163 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:42.407337 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:42.408727 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:42.409402 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:42.409561 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:42.412768 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:42.412776 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:42.412782 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:42.412839 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:42.417060 (+ 4221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:42.418288 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:42.418294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:42.418297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:42.418528 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:42.419102 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:42.419107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:42.419110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:42.419662 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:42.420024 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:42.420523 (+ 499us) server_negotiation.cc:299] Negotiation successful
0928 16:55:42.420785 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:55:42.425843 31823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:55:42.436818 31823 catalog_manager.cc:1347] Generated new cluster ID: 956893a13605457dacbda650c7ee5c39
I20260928 16:55:42.436956 31823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:55:42.452804 31823 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:55:42.454370 31823 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:55:42.462507 31823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcfec095d6df49c8ab4227bea915373e: Generated new TSK 0
I20260928 16:55:42.463617 31823 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:55:42.469880 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 31742
I20260928 16:55:42.471215 31804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:55:42.563974 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 31742
2026-09-28T16:55:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2063 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:55:42 dist-test-slave-wzcb krb5kdc[31841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:55:42 dist-test-slave-wzcb krb5kdc[31841](info): set up 2 sockets
Sep 28 16:55:42 dist-test-slave-wzcb krb5kdc[31841](info): commencing operation
krb5kdc: starting...
W20260928 16:55:43.614256 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:55:43 dist-test-slave-wzcb krb5kdc[31841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:55:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:55:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:55:43.930745 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:42009
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:42009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:55:44.311376 31854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:44.311764 31854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:44.311909 31854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:44.319732 31854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:44.319895 31854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:44.319985 31854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:44.320072 31854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:44.320132 31854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:44.331049 31854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:42009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:42009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:44.332672 31854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:44.334206 31854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:44.347546 31860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:44.347622 31859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:44.347582 31862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:44.348603 31854 server_base.cc:1034] running on GCE node
I20260928 16:55:44.349529 31854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:44.351217 31854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:44.352509 31854 hybrid_clock.cc:648] HybridClock initialized: now 1790614544352434 us; error 54 us; skew 500 ppm
I20260928 16:55:44.352914 31854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:44 dist-test-slave-wzcb krb5kdc[31841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614544, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:44.360184 31854 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:55:44.362185 31854 webserver.cc:460] Webserver started at http://127.9.93.62:37675/ using document root <none> and password file <none>
I20260928 16:55:44.363118 31854 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:44.363277 31854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:44.363654 31854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:44.367297 31854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "9f476118ca7c417a92eff7d9914c30ff"
format_stamp: "Formatted at 2026-09-28 16:55:44 on dist-test-slave-wzcb"
server_key: "e9e5a7e5f94dd53c007e4d2880b3411c"
server_key_iv: "b0051b7a5443353ad8df9bcb51a47434"
server_key_version: "encryptionkey@0"
I20260928 16:55:44.368332 31854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "9f476118ca7c417a92eff7d9914c30ff"
format_stamp: "Formatted at 2026-09-28 16:55:44 on dist-test-slave-wzcb"
server_key: "e9e5a7e5f94dd53c007e4d2880b3411c"
server_key_iv: "b0051b7a5443353ad8df9bcb51a47434"
server_key_version: "encryptionkey@0"
I20260928 16:55:44.374871 31854 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 16:55:44.379436 31869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:44.381283 31854 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:55:44.381541 31854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "9f476118ca7c417a92eff7d9914c30ff"
format_stamp: "Formatted at 2026-09-28 16:55:44 on dist-test-slave-wzcb"
server_key: "e9e5a7e5f94dd53c007e4d2880b3411c"
server_key_iv: "b0051b7a5443353ad8df9bcb51a47434"
server_key_version: "encryptionkey@0"
I20260928 16:55:44.381834 31854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:44.408195 31854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:44.413458 31854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:44.413806 31854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:44.433558 31854 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:42009
I20260928 16:55:44.433599 31920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:42009 every 8 connection(s)
I20260928 16:55:44.435164 31854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:55:44.439965 31921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:44.444584 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 31854
I20260928 16:55:44.444824 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:55:44.445494 9588 external_mini_cluster.cc:1428] Setting key c3cf8dcfd367ff162a546702aa996b36
I20260928 16:55:44.455176 31921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f476118ca7c417a92eff7d9914c30ff: Bootstrap starting.
I20260928 16:55:44.459420 31921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f476118ca7c417a92eff7d9914c30ff: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:44.460832 31921 log.cc:824] T 00000000000000000000000000000000 P 9f476118ca7c417a92eff7d9914c30ff: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:44.460990 31924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:44.447866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59817 (local address 127.9.93.62:42009)
0928 16:55:44.448778 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:44.448792 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:44.448841 (+ 49us) server_negotiation.cc:407] Connection header received
0928 16:55:44.450669 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:44.450698 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:44.451217 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:44.451683 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:44.452416 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:44.453926 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:44.455871 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:44.456660 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:44.456810 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:44.458301 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:44.458310 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:44.458328 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:44.458945 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:44.459045 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:44.459073 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:44.459170 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:44.459484 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 16:55:44.459792 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":438,"threads_started":1}
I20260928 16:55:44.464233 31921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f476118ca7c417a92eff7d9914c30ff: No bootstrap required, opened a new log
I20260928 16:55:44.465222 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:55:44.473842 31921 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f476118ca7c417a92eff7d9914c30ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f476118ca7c417a92eff7d9914c30ff" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42009 } }
I20260928 16:55:44.474485 31921 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f476118ca7c417a92eff7d9914c30ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:44.474625 31921 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f476118ca7c417a92eff7d9914c30ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f476118ca7c417a92eff7d9914c30ff, State: Initialized, Role: FOLLOWER
I20260928 16:55:44.475472 31921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f476118ca7c417a92eff7d9914c30ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f476118ca7c417a92eff7d9914c30ff" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42009 } }
I20260928 16:55:44.475823 31921 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f476118ca7c417a92eff7d9914c30ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:55:44.475981 31921 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f476118ca7c417a92eff7d9914c30ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:55:44.476197 31921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f476118ca7c417a92eff7d9914c30ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:44.477434 31924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:44.470014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55189 (local address 127.9.93.62:42009)
0928 16:55:44.470283 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:44.470290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:44.470594 (+ 304us) server_negotiation.cc:407] Connection header received
0928 16:55:44.470793 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:44.470797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:44.470871 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:44.470958 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:44.471702 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:44.472585 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:44.474492 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:44.475048 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:44.475156 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:44.476518 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:44.476524 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:44.476534 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:44.476575 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:44.476647 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:44.476666 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:44.476753 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:44.476929 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 16:55:44.477036 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 16:55:44.479274 31921 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f476118ca7c417a92eff7d9914c30ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f476118ca7c417a92eff7d9914c30ff" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42009 } }
I20260928 16:55:44.479871 31921 leader_election.cc:302] T 00000000000000000000000000000000 P 9f476118ca7c417a92eff7d9914c30ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f476118ca7c417a92eff7d9914c30ff; no voters:
I20260928 16:55:44.480522 31921 leader_election.cc:288] T 00000000000000000000000000000000 P 9f476118ca7c417a92eff7d9914c30ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:55:44.480702 31931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f476118ca7c417a92eff7d9914c30ff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:55:44.481416 31931 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f476118ca7c417a92eff7d9914c30ff [term 1 LEADER]: Becoming Leader. State: Replica: 9f476118ca7c417a92eff7d9914c30ff, State: Running, Role: LEADER
I20260928 16:55:44.482106 31931 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f476118ca7c417a92eff7d9914c30ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f476118ca7c417a92eff7d9914c30ff" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42009 } }
I20260928 16:55:44.482972 31921 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f476118ca7c417a92eff7d9914c30ff [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:55:44.485267 31932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f476118ca7c417a92eff7d9914c30ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f476118ca7c417a92eff7d9914c30ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f476118ca7c417a92eff7d9914c30ff" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42009 } } }
I20260928 16:55:44.485687 31932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f476118ca7c417a92eff7d9914c30ff [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:44.486189 31933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f476118ca7c417a92eff7d9914c30ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f476118ca7c417a92eff7d9914c30ff. Latest consensus state: current_term: 1 leader_uuid: "9f476118ca7c417a92eff7d9914c30ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f476118ca7c417a92eff7d9914c30ff" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 42009 } } }
I20260928 16:55:44.486471 31933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f476118ca7c417a92eff7d9914c30ff [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:44.490221 31939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:55:44.496833 31939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:55:44.506948 31939 catalog_manager.cc:1347] Generated new cluster ID: 813673ecf10a46a898b1175b72b8711f
I20260928 16:55:44.507107 31939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:55:44.517498 31939 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:55:44.519714 31939 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:55:44.531270 31939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f476118ca7c417a92eff7d9914c30ff: Generated new TSK 0
I20260928 16:55:44.532689 31939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:55:44.549432 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 31854
W20260928 16:55:44.550810 31916 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59817
W20260928 16:55:44.551663 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 31854: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:55:44.551847 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 31854
2026-09-28T16:55:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1988 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:55:44 dist-test-slave-wzcb krb5kdc[31953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:55:44 dist-test-slave-wzcb krb5kdc[31953](info): set up 2 sockets
Sep 28 16:55:44 dist-test-slave-wzcb krb5kdc[31953](info): commencing operation
krb5kdc: starting...
W20260928 16:55:45.606771 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:55:45 dist-test-slave-wzcb krb5kdc[31953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:55:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:55:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:55:45.930445 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:34301
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:42669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:34301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:55:46.311503 31966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:46.311870 31966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:46.312004 31966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:46.319944 31966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:46.320089 31966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:46.320150 31966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:46.320245 31966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:46.320300 31966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:46.331224 31966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:42669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:34301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:34301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:46.332844 31966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:46.334393 31966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:46.347766 31971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:46.347834 31972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:46.347965 31974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:46.348766 31966 server_base.cc:1034] running on GCE node
I20260928 16:55:46.349767 31966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:46.351576 31966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:46.352826 31966 hybrid_clock.cc:648] HybridClock initialized: now 1790614546352751 us; error 73 us; skew 500 ppm
I20260928 16:55:46.353272 31966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:46 dist-test-slave-wzcb krb5kdc[31953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614546, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:46.357007 31966 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:55:46.358764 31966 webserver.cc:460] Webserver started at http://127.9.93.62:35441/ using document root <none> and password file <none>
I20260928 16:55:46.359664 31966 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:46.359804 31966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:46.360137 31966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:46.363659 31966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "dddb347b2ff74eb1a1f242d14d9ee238"
format_stamp: "Formatted at 2026-09-28 16:55:46 on dist-test-slave-wzcb"
server_key: "c8e4a26dbfe1d6c1a6c7a294fcd85881"
server_key_iv: "8e1490f1969b7ba0325506d84912e56b"
server_key_version: "encryptionkey@0"
I20260928 16:55:46.364519 31966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "dddb347b2ff74eb1a1f242d14d9ee238"
format_stamp: "Formatted at 2026-09-28 16:55:46 on dist-test-slave-wzcb"
server_key: "c8e4a26dbfe1d6c1a6c7a294fcd85881"
server_key_iv: "8e1490f1969b7ba0325506d84912e56b"
server_key_version: "encryptionkey@0"
I20260928 16:55:46.370667 31966 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 16:55:46.374987 31982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:46.376729 31966 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:55:46.376968 31966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "dddb347b2ff74eb1a1f242d14d9ee238"
format_stamp: "Formatted at 2026-09-28 16:55:46 on dist-test-slave-wzcb"
server_key: "c8e4a26dbfe1d6c1a6c7a294fcd85881"
server_key_iv: "8e1490f1969b7ba0325506d84912e56b"
server_key_version: "encryptionkey@0"
I20260928 16:55:46.377199 31966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:46.383786 31966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:46.388612 31966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:46.388953 31966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:46.408643 31966 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:34301
I20260928 16:55:46.408687 32035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:34301 every 8 connection(s)
I20260928 16:55:46.410208 31966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:55:46.414577 32036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:46.414642 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 31966
I20260928 16:55:46.414819 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:55:46.415254 9588 external_mini_cluster.cc:1428] Setting key e2ce884795cbfceb8ced88bed6f272ab
I20260928 16:55:46.423863 32039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:46.417192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58407 (local address 127.9.93.62:34301)
0928 16:55:46.418096 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:46.418110 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:46.418162 (+ 52us) server_negotiation.cc:407] Connection header received
0928 16:55:46.419961 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:46.419990 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:46.420410 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:46.420807 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:46.422046 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:46.422053 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:46.422069 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:46.422087 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:46.422159 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:46.422185 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:46.422378 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:46.422485 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 16:55:46.422700 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":389,"threads_started":1}
I20260928 16:55:46.428260 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:55:46.429618 32036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dddb347b2ff74eb1a1f242d14d9ee238: Bootstrap starting.
I20260928 16:55:46.433476 32036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dddb347b2ff74eb1a1f242d14d9ee238: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:46.435047 32036 log.cc:824] T 00000000000000000000000000000000 P dddb347b2ff74eb1a1f242d14d9ee238: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:46.435607 32039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:46.432843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51217 (local address 127.9.93.62:34301)
0928 16:55:46.433063 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:46.433069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:46.433438 (+ 369us) server_negotiation.cc:407] Connection header received
0928 16:55:46.433688 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:46.433692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:46.433750 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:46.433839 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:46.434861 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:46.434865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:46.434868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:55:46.434902 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:46.434943 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:55:46.434959 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:46.435054 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:46.435212 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 16:55:46.435279 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 16:55:46.438432 32036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dddb347b2ff74eb1a1f242d14d9ee238: No bootstrap required, opened a new log
I20260928 16:55:46.446389 32036 raft_consensus.cc:348] T 00000000000000000000000000000000 P dddb347b2ff74eb1a1f242d14d9ee238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dddb347b2ff74eb1a1f242d14d9ee238" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 34301 } }
I20260928 16:55:46.446854 32036 raft_consensus.cc:374] T 00000000000000000000000000000000 P dddb347b2ff74eb1a1f242d14d9ee238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:55:46.446982 32036 raft_consensus.cc:729] T 00000000000000000000000000000000 P dddb347b2ff74eb1a1f242d14d9ee238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dddb347b2ff74eb1a1f242d14d9ee238, State: Initialized, Role: FOLLOWER
I20260928 16:55:46.447666 32036 consensus_queue.cc:260] T 00000000000000000000000000000000 P dddb347b2ff74eb1a1f242d14d9ee238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dddb347b2ff74eb1a1f242d14d9ee238" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 34301 } }
I20260928 16:55:46.447912 32036 raft_consensus.cc:388] T 00000000000000000000000000000000 P dddb347b2ff74eb1a1f242d14d9ee238 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:55:46.448052 32036 raft_consensus.cc:482] T 00000000000000000000000000000000 P dddb347b2ff74eb1a1f242d14d9ee238 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:55:46.448212 32036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dddb347b2ff74eb1a1f242d14d9ee238 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:46.450603 32036 raft_consensus.cc:504] T 00000000000000000000000000000000 P dddb347b2ff74eb1a1f242d14d9ee238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dddb347b2ff74eb1a1f242d14d9ee238" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 34301 } }
I20260928 16:55:46.451195 32036 leader_election.cc:302] T 00000000000000000000000000000000 P dddb347b2ff74eb1a1f242d14d9ee238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dddb347b2ff74eb1a1f242d14d9ee238; no voters:
I20260928 16:55:46.451870 32036 leader_election.cc:288] T 00000000000000000000000000000000 P dddb347b2ff74eb1a1f242d14d9ee238 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:55:46.452049 32046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dddb347b2ff74eb1a1f242d14d9ee238 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:55:46.453071 32046 raft_consensus.cc:686] T 00000000000000000000000000000000 P dddb347b2ff74eb1a1f242d14d9ee238 [term 1 LEADER]: Becoming Leader. State: Replica: dddb347b2ff74eb1a1f242d14d9ee238, State: Running, Role: LEADER
I20260928 16:55:46.453768 32046 consensus_queue.cc:237] T 00000000000000000000000000000000 P dddb347b2ff74eb1a1f242d14d9ee238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dddb347b2ff74eb1a1f242d14d9ee238" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 34301 } }
I20260928 16:55:46.454295 32036 sys_catalog.cc:564] T 00000000000000000000000000000000 P dddb347b2ff74eb1a1f242d14d9ee238 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:55:46.456737 32048 sys_catalog.cc:455] T 00000000000000000000000000000000 P dddb347b2ff74eb1a1f242d14d9ee238 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dddb347b2ff74eb1a1f242d14d9ee238. Latest consensus state: current_term: 1 leader_uuid: "dddb347b2ff74eb1a1f242d14d9ee238" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dddb347b2ff74eb1a1f242d14d9ee238" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 34301 } } }
I20260928 16:55:46.457018 32047 sys_catalog.cc:455] T 00000000000000000000000000000000 P dddb347b2ff74eb1a1f242d14d9ee238 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dddb347b2ff74eb1a1f242d14d9ee238" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dddb347b2ff74eb1a1f242d14d9ee238" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 34301 } } }
I20260928 16:55:46.457233 32048 sys_catalog.cc:458] T 00000000000000000000000000000000 P dddb347b2ff74eb1a1f242d14d9ee238 [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:46.457276 32047 sys_catalog.cc:458] T 00000000000000000000000000000000 P dddb347b2ff74eb1a1f242d14d9ee238 [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:46.459743 32055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:55:46.465615 32055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:55:46.477221 32055 catalog_manager.cc:1347] Generated new cluster ID: 2bea34454f714bd18a32633c6b4cca5b
I20260928 16:55:46.477348 32055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:55:46.495330 32055 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:55:46.495980 32055 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:55:46.503996 32055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dddb347b2ff74eb1a1f242d14d9ee238: Generated new TSK 0
I20260928 16:55:46.505357 32055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:55:46.524772 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 31966
W20260928 16:55:46.526010 32031 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58407
W20260928 16:55:46.526767 9588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 31966: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:55:46.526963 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 31966
2026-09-28T16:55:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1974 ms)
[----------] 18 tests from AuthTokenIssuingTest (36354 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:55:46 dist-test-slave-wzcb krb5kdc[32068](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:55:46 dist-test-slave-wzcb krb5kdc[32068](info): set up 2 sockets
Sep 28 16:55:46 dist-test-slave-wzcb krb5kdc[32068](info): commencing operation
krb5kdc: starting...
W20260928 16:55:47.578428 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:55:47 dist-test-slave-wzcb krb5kdc[32068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:55:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:55:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:55:47.903079 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:46549
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:39429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:46549,127.9.93.61:37681,127.9.93.60:46753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:55:48.294008 32081 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:55:48.294389 32081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:48.294512 32081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:48.294634 32081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:48.302716 32081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:48.302840 32081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:48.302913 32081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:48.302989 32081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:48.303081 32081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:48.314055 32081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:46549,127.9.93.61:37681,127.9.93.60:46753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:46549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:48.315817 32081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:48.317365 32081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:48.330149 32087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:48.330212 32089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:48.330188 32086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:48.331187 32081 server_base.cc:1034] running on GCE node
I20260928 16:55:48.332165 32081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:48.333905 32081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:48.335157 32081 hybrid_clock.cc:648] HybridClock initialized: now 1790614548335076 us; error 53 us; skew 500 ppm
I20260928 16:55:48.335579 32081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:48 dist-test-slave-wzcb krb5kdc[32068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614548, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:48.339165 32081 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:55:48.340948 32081 webserver.cc:460] Webserver started at http://127.9.93.62:46379/ using document root <none> and password file <none>
I20260928 16:55:48.341753 32081 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:48.341862 32081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:48.342197 32081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:48.345742 32081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "513846aa62c8459980abc7da7584a959"
format_stamp: "Formatted at 2026-09-28 16:55:48 on dist-test-slave-wzcb"
server_key: "3825710be4cb8abab93e6289686a8e2a"
server_key_iv: "95851c550464bb897eb8a14d229fa4fc"
server_key_version: "encryptionkey@0"
I20260928 16:55:48.346661 32081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "513846aa62c8459980abc7da7584a959"
format_stamp: "Formatted at 2026-09-28 16:55:48 on dist-test-slave-wzcb"
server_key: "3825710be4cb8abab93e6289686a8e2a"
server_key_iv: "95851c550464bb897eb8a14d229fa4fc"
server_key_version: "encryptionkey@0"
I20260928 16:55:48.353348 32081 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 16:55:48.357726 32096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:48.359474 32081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:55:48.359674 32081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "513846aa62c8459980abc7da7584a959"
format_stamp: "Formatted at 2026-09-28 16:55:48 on dist-test-slave-wzcb"
server_key: "3825710be4cb8abab93e6289686a8e2a"
server_key_iv: "95851c550464bb897eb8a14d229fa4fc"
server_key_version: "encryptionkey@0"
I20260928 16:55:48.359920 32081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:48.386824 32081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:48.391695 32081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:48.391997 32081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:48.411756 32081 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:46549
I20260928 16:55:48.411763 32148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:46549 every 8 connection(s)
I20260928 16:55:48.413364 32081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:55:48.417054 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 32081
I20260928 16:55:48.417279 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:55:48.417910 32149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:48.417994 9588 external_mini_cluster.cc:1428] Setting key 120f5b21cee1a090931448a34240a400
I20260928 16:55:48.425539 32149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46549 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:55:48 dist-test-slave-wzcb krb5kdc[32068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614548, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.93.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.61@KRBTEST.COM" created.
I20260928 16:55:48.450841 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:48.429391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:46549 (local address 127.0.0.1:42882)
0928 16:55:48.430656 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:48.430709 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:48.432716 (+ 2007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:48.433746 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:48.433754 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:48.434576 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:48.434971 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:48.434992 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:48.436715 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:48.436719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:48.437901 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:48.437914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:48.438954 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:48.438961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:48.439144 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:48.439823 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:55:48.439848 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:48.442536 (+ 2688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:48.446153 (+ 3617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:48.446167 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:48.446179 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:48.446698 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:48.447194 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:48.447200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:48.447204 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:48.447561 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:48.448502 (+ 941us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:48.448509 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:48.448756 (+ 247us) client_negotiation.cc:769] Sending connection context
0928 16:55:48.448993 (+ 237us) client_negotiation.cc:240] Negotiation successful
0928 16:55:48.449551 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"mutex_wait_us":3,"thread_start_us":805,"threads_started":1}
I20260928 16:55:48.450870 32152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:48.429390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42882 (local address 127.9.93.62:46549)
0928 16:55:48.430319 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:48.430339 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:48.430843 (+ 504us) server_negotiation.cc:407] Connection header received
0928 16:55:48.433184 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:48.433191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:48.433491 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:48.433695 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:48.435200 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:48.436549 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:48.438050 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:48.438681 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:48.438882 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:48.442867 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:48.442877 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:48.442894 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:48.442931 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:48.445962 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:48.446844 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:48.446852 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:48.446855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:48.447045 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:48.447678 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:48.447682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:48.447684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:48.448347 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:48.448556 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:48.449185 (+ 629us) server_negotiation.cc:299] Negotiation successful
0928 16:55:48.449712 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":205,"server-negotiator.queue_time_us":653,"thread_start_us":400,"threads_started":1}
I20260928 16:55:48.454721 32149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:55:48.456082 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:48.455640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:37681 (local address 127.0.0.1:42864)
0928 16:55:48.455825 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:48.455949 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:37681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 16:55:48.456431 32100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.93.61:37681: connect: Connection refused (error 111)
I20260928 16:55:48.458205 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:48.457785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:37681 (local address 127.0.0.1:42868)
0928 16:55:48.457944 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:48.458043 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:37681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 16:55:48.458446 32149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.61:37681: Network error: Client connection negotiation failed: client connection to 127.9.93.61:37681: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.61.keytab.
Entry for principal kudu/127.9.93.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.61.keytab.
I20260928 16:55:48.487185 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.61:37681
--webserver_interface=127.9.93.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.61.keytab
--principal=kudu/127.9.93.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:39429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:46549,127.9.93.61:37681,127.9.93.60:46753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:55:48.507901 32149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } attempt: 1
I20260928 16:55:48.509305 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:48.508752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:37681 (local address 127.0.0.1:42882)
0928 16:55:48.508991 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:48.509130 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:37681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 16:55:48.510510 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:48.510039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:37681 (local address 127.0.0.1:42884)
0928 16:55:48.510219 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:48.510320 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:37681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 16:55:48.510711 32149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.61:37681: Network error: Client connection negotiation failed: client connection to 127.9.93.61:37681: connect: Connection refused (error 111)
I20260928 16:55:48.579034 32149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } attempt: 2
I20260928 16:55:48.580446 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:48.579841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:37681 (local address 127.0.0.1:42890)
0928 16:55:48.580123 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:48.580249 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:37681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 16:55:48.581573 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:48.581158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:37681 (local address 127.0.0.1:42894)
0928 16:55:48.581339 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:48.581425 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:37681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 16:55:48.581796 32149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.61:37681: Network error: Client connection negotiation failed: client connection to 127.9.93.61:37681: connect: Connection refused (error 111)
I20260928 16:55:48.673079 32149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } attempt: 3
I20260928 16:55:48.674295 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:48.673807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:37681 (local address 127.0.0.1:42902)
0928 16:55:48.674043 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:48.674141 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:37681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 16:55:48.675410 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:48.674972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:37681 (local address 127.0.0.1:42910)
0928 16:55:48.675125 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:48.675201 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:37681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 16:55:48.675624 32149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.61:37681: Network error: Client connection negotiation failed: client connection to 127.9.93.61:37681: connect: Connection refused (error 111)
I20260928 16:55:48.818905 32149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } attempt: 4
I20260928 16:55:48.820422 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:48.819707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:37681 (local address 127.0.0.1:42920)
0928 16:55:48.820159 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:48.820269 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:37681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259}
I20260928 16:55:48.821614 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:48.821182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:37681 (local address 127.0.0.1:42926)
0928 16:55:48.821348 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:48.821448 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:37681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 16:55:48.821851 32149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.61:37681: Network error: Client connection negotiation failed: client connection to 127.9.93.61:37681: connect: Connection refused (error 111)
W20260928 16:55:48.860735 32156 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:55:48.861143 32156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:48.861245 32156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:48.861370 32156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:48.869607 32156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:48.869735 32156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:48.869818 32156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:48.869885 32156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:48.869971 32156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:48.881134 32156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:46549,127.9.93.61:37681,127.9.93.60:46753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.61.keytab
--principal=kudu/127.9.93.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.61:37681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.93.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:48.882692 32156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:48.884298 32156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:48.897745 32161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:48.897660 32162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:48.897706 32164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:48.898674 32156 server_base.cc:1034] running on GCE node
I20260928 16:55:48.899745 32156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:48.901477 32156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:48.902762 32156 hybrid_clock.cc:648] HybridClock initialized: now 1790614548902660 us; error 67 us; skew 500 ppm
I20260928 16:55:48.903190 32156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:48 dist-test-slave-wzcb krb5kdc[32068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614548, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:48.907141 32156 init.cc:377] Logged in from keytab as kudu/127.9.93.61@KRBTEST.COM (short username kudu)
I20260928 16:55:48.909189 32156 webserver.cc:460] Webserver started at http://127.9.93.61:34383/ using document root <none> and password file <none>
I20260928 16:55:48.910063 32156 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:48.910192 32156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:48.910567 32156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:48.914289 32156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-1/data/instance:
uuid: "cff6e6ea6e6e48089b71aebe4cffbdc5"
format_stamp: "Formatted at 2026-09-28 16:55:48 on dist-test-slave-wzcb"
server_key: "bc8d1ed3627a13fd4f7c3d2fd7ee7e68"
server_key_iv: "6dcd4cfdd812bb434f8e8bf7075f0e32"
server_key_version: "encryptionkey@0"
I20260928 16:55:48.915290 32156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-1/wal/instance:
uuid: "cff6e6ea6e6e48089b71aebe4cffbdc5"
format_stamp: "Formatted at 2026-09-28 16:55:48 on dist-test-slave-wzcb"
server_key: "bc8d1ed3627a13fd4f7c3d2fd7ee7e68"
server_key_iv: "6dcd4cfdd812bb434f8e8bf7075f0e32"
server_key_version: "encryptionkey@0"
I20260928 16:55:48.921936 32156 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:55:48.926455 32171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:48.928354 32156 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 16:55:48.928596 32156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-1/wal
uuid: "cff6e6ea6e6e48089b71aebe4cffbdc5"
format_stamp: "Formatted at 2026-09-28 16:55:48 on dist-test-slave-wzcb"
server_key: "bc8d1ed3627a13fd4f7c3d2fd7ee7e68"
server_key_iv: "6dcd4cfdd812bb434f8e8bf7075f0e32"
server_key_version: "encryptionkey@0"
I20260928 16:55:48.928879 32156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:48.949116 32156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:48.954293 32156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:48.954617 32156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:48.975132 32156 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.61:37681
I20260928 16:55:48.975152 32223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.61:37681 every 8 connection(s)
I20260928 16:55:48.976775 32156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-1/data/info.pb
I20260928 16:55:48.980499 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 32156
I20260928 16:55:48.980712 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-1/wal/instance
I20260928 16:55:48.981204 9588 external_mini_cluster.cc:1428] Setting key 96a734f9485039d765561705fdc45442
I20260928 16:55:48.981611 32224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:48.989354 32224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46549 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:55:49 dist-test-slave-wzcb krb5kdc[32068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614548, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.61@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.93.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.60@KRBTEST.COM" created.
I20260928 16:55:49.017426 32227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:48.992672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42892 (local address 127.9.93.62:46549)
0928 16:55:48.993227 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:48.993234 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:48.994604 (+ 1370us) server_negotiation.cc:407] Connection header received
0928 16:55:48.997715 (+ 3111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:48.997721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:48.997818 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:48.997948 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:49.000795 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.001612 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.004251 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.004712 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.004830 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:49.010005 (+ 5175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:49.010012 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:49.010016 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:49.010074 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:49.013287 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:49.014256 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:49.014264 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:49.014268 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:49.014428 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:49.014917 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:49.014922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:49.014924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:49.015419 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:49.015587 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:49.016706 (+ 1119us) server_negotiation.cc:299] Negotiation successful
0928 16:55:49.017025 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":274,"threads_started":1}
I20260928 16:55:49.018512 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:48.993431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:46549 (local address 127.0.0.1:42892)
0928 16:55:48.994376 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:48.994446 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:48.997215 (+ 2769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:48.998349 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:48.998363 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:48.999247 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:49.000408 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:49.000441 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.001853 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.001858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:49.004054 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:49.004075 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.005016 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.005024 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:49.005364 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:49.006220 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:55:49.006261 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:49.009599 (+ 3338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:49.013489 (+ 3890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:49.013500 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:49.013517 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:49.014095 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:49.014555 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:49.014559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:49.014561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:49.014792 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:49.015596 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:49.015604 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:49.016180 (+ 576us) client_negotiation.cc:769] Sending connection context
0928 16:55:49.016525 (+ 345us) client_negotiation.cc:240] Negotiation successful
0928 16:55:49.017019 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":361,"threads_started":1}
I20260928 16:55:49.020659 32224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:55:49 dist-test-slave-wzcb krb5kdc[32068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614548, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.61@KRBTEST.COM for kudu/127.9.93.61@KRBTEST.COM
I20260928 16:55:49.038517 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.021625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:37681 (local address 127.0.0.1:42928)
0928 16:55:49.021849 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.021869 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:49.021982 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:49.023420 (+ 1438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:49.023425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:49.023921 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:49.024074 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:49.024084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.025587 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.025592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:49.026673 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:49.026690 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.027806 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.027812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:49.027994 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:49.028744 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:55:49.028766 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:49.031424 (+ 2658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:49.035367 (+ 3943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:49.035372 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:49.035377 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:49.035975 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:49.036500 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:49.036505 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:49.036509 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:49.036675 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:49.037424 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:49.037429 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:49.037605 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 16:55:49.037794 (+ 189us) client_negotiation.cc:240] Negotiation successful
0928 16:55:49.038150 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 16:55:49.038827 32229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.022077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42928 (local address 127.9.93.61:37681)
0928 16:55:49.022867 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:49.022874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:49.022905 (+ 31us) server_negotiation.cc:407] Connection header received
0928 16:55:49.022963 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:49.022970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:49.023204 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:49.023419 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:49.024256 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.025410 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.026861 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.027630 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.027759 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:49.031683 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:49.031702 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:49.031720 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:49.031767 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:49.035139 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:49.036175 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:49.036184 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:49.036187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:49.036355 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:49.036805 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:49.036809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:49.036811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:49.037277 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:49.037488 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:49.037986 (+ 498us) server_negotiation.cc:299] Negotiation successful
0928 16:55:49.038305 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":448,"threads_started":1}
I20260928 16:55:49.041743 32224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:55:49.043197 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.042523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:46753 (local address 127.0.0.1:59476)
0928 16:55:49.042764 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.042953 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260928 16:55:49.043609 32172 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
I20260928 16:55:49.045507 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.044997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:46753 (local address 127.0.0.1:59486)
0928 16:55:49.045203 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.045346 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260928 16:55:49.045790 32224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:46753: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.60.keytab.
Entry for principal kudu/127.9.93.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.60.keytab.
I20260928 16:55:49.055564 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.60:46753
--webserver_interface=127.9.93.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.60.keytab
--principal=kudu/127.9.93.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:39429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:46549,127.9.93.61:37681,127.9.93.60:46753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:55:49.095224 32224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } attempt: 1
I20260928 16:55:49.096662 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.096088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:46753 (local address 127.0.0.1:59494)
0928 16:55:49.096326 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.096452 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 16:55:49.097867 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.097432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:46753 (local address 127.0.0.1:59506)
0928 16:55:49.097603 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.097712 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 16:55:49.098115 32224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:46753: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
I20260928 16:55:49.121143 32149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } attempt: 5
Sep 28 16:55:49 dist-test-slave-wzcb krb5kdc[32068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614548, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for kudu/127.9.93.61@KRBTEST.COM
I20260928 16:55:49.137668 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.122072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:37681 (local address 127.0.0.1:42944)
0928 16:55:49.122314 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.122339 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:49.122518 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:49.122919 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:49.122923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:49.123435 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:49.123653 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:49.123664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.125151 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.125157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:49.126270 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:49.126288 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.127434 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.127440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:49.127640 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:49.128286 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:55:49.128305 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:49.130717 (+ 2412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:49.134915 (+ 4198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:49.134921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:49.134924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:49.135482 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:49.135988 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:49.135992 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:49.135994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:49.136151 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:49.136813 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:49.136817 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:49.136974 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 16:55:49.137095 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 16:55:49.137392 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 16:55:49.137840 32229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.122205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42944 (local address 127.9.93.61:37681)
0928 16:55:49.122370 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:49.122376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:49.122460 (+ 84us) server_negotiation.cc:407] Connection header received
0928 16:55:49.122646 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:49.122652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:49.122781 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:49.122897 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:49.123822 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.124922 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.126449 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.127182 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.127357 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:49.131002 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:49.131009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:49.131014 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:49.131067 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:49.134678 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:49.135627 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:49.135633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:49.135636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:49.135823 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:49.136275 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:49.136280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:49.136283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:49.136685 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:49.136851 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:49.137249 (+ 398us) server_negotiation.cc:299] Negotiation successful
0928 16:55:49.137510 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:55:49.138993 32149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:55:49.140326 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.139825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:46753 (local address 127.0.0.1:59520)
0928 16:55:49.140041 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.140142 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 16:55:49.141404 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.141015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:46753 (local address 127.0.0.1:59526)
0928 16:55:49.141170 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.141254 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 16:55:49.141654 32149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:46753: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
I20260928 16:55:49.166378 32224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } attempt: 2
I20260928 16:55:49.167817 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.167196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:46753 (local address 127.0.0.1:59532)
0928 16:55:49.167461 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.167593 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 16:55:49.169119 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.168667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:46753 (local address 127.0.0.1:59538)
0928 16:55:49.168841 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.168945 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 16:55:49.169422 32224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:46753: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
I20260928 16:55:49.192946 32149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } attempt: 1
I20260928 16:55:49.194335 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.193760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:46753 (local address 127.0.0.1:59552)
0928 16:55:49.194011 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.194116 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 16:55:49.195621 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.195125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:46753 (local address 127.0.0.1:59558)
0928 16:55:49.195362 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.195460 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260928 16:55:49.195914 32149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:46753: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
I20260928 16:55:49.260731 32224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } attempt: 3
I20260928 16:55:49.262318 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.261678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:46753 (local address 127.0.0.1:59562)
0928 16:55:49.261962 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.262102 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260928 16:55:49.263674 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.263083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:46753 (local address 127.0.0.1:59570)
0928 16:55:49.263374 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.263481 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260928 16:55:49.263938 32224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:46753: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
I20260928 16:55:49.264143 32149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } attempt: 2
I20260928 16:55:49.265266 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.264760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:46753 (local address 127.0.0.1:59580)
0928 16:55:49.264963 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.265087 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 16:55:49.266274 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.265951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:46753 (local address 127.0.0.1:59590)
0928 16:55:49.266075 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.266147 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 16:55:49.266476 32149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:46753: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
I20260928 16:55:49.372709 32149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } attempt: 3
I20260928 16:55:49.374177 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.373549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:46753 (local address 127.0.0.1:59600)
0928 16:55:49.373816 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.373973 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 16:55:49.375816 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.375099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:46753 (local address 127.0.0.1:59610)
0928 16:55:49.375500 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.375624 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
W20260928 16:55:49.376066 32149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:46753: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
I20260928 16:55:49.407202 32224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } attempt: 4
I20260928 16:55:49.408545 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.407994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:46753 (local address 127.0.0.1:59616)
0928 16:55:49.408269 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.408372 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 16:55:49.409837 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.409375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:46753 (local address 127.0.0.1:59632)
0928 16:55:49.409575 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.409681 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260928 16:55:49.410112 32224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:46753: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
W20260928 16:55:49.459694 32232 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:55:49.460067 32232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:49.460142 32232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:49.460230 32232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:49.468142 32232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:49.468276 32232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:49.468343 32232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:49.468395 32232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:49.468459 32232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:49.479086 32232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:39429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:46549,127.9.93.61:37681,127.9.93.60:46753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.60.keytab
--principal=kudu/127.9.93.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.60:46753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.93.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:49.480702 32232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:49.482182 32232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:49.495321 32241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:49.495198 32238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:49.495326 32239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:49.496227 32232 server_base.cc:1034] running on GCE node
I20260928 16:55:49.497149 32232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:49.498867 32232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:49.500123 32232 hybrid_clock.cc:648] HybridClock initialized: now 1790614549500025 us; error 69 us; skew 500 ppm
I20260928 16:55:49.500502 32232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:49 dist-test-slave-wzcb krb5kdc[32068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614549, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:49.507289 32232 init.cc:377] Logged in from keytab as kudu/127.9.93.60@KRBTEST.COM (short username kudu)
I20260928 16:55:49.509231 32232 webserver.cc:460] Webserver started at http://127.9.93.60:40569/ using document root <none> and password file <none>
I20260928 16:55:49.510361 32232 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:49.510488 32232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:49.510816 32232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:49.514386 32232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-2/data/instance:
uuid: "20673971547741f6aab0f8fdd9b28b6e"
format_stamp: "Formatted at 2026-09-28 16:55:49 on dist-test-slave-wzcb"
server_key: "28bad0b039cb7f1f91da93376d499041"
server_key_iv: "4a5613f07d64d5a342a5e9862d32b76a"
server_key_version: "encryptionkey@0"
I20260928 16:55:49.515358 32232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-2/wal/instance:
uuid: "20673971547741f6aab0f8fdd9b28b6e"
format_stamp: "Formatted at 2026-09-28 16:55:49 on dist-test-slave-wzcb"
server_key: "28bad0b039cb7f1f91da93376d499041"
server_key_iv: "4a5613f07d64d5a342a5e9862d32b76a"
server_key_version: "encryptionkey@0"
I20260928 16:55:49.521631 32232 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:55:49.525990 32248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:49.527976 32232 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:55:49.528177 32232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-2/wal
uuid: "20673971547741f6aab0f8fdd9b28b6e"
format_stamp: "Formatted at 2026-09-28 16:55:49 on dist-test-slave-wzcb"
server_key: "28bad0b039cb7f1f91da93376d499041"
server_key_iv: "4a5613f07d64d5a342a5e9862d32b76a"
server_key_version: "encryptionkey@0"
I20260928 16:55:49.528409 32232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:49.544858 32232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:49.550197 32232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:49.550527 32232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:49.553346 32149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } attempt: 4
I20260928 16:55:49.554638 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.554107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:46753 (local address 127.0.0.1:59648)
0928 16:55:49.554367 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.554472 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 16:55:49.555855 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.555455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:46753 (local address 127.0.0.1:59660)
0928 16:55:49.555620 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.555707 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 16:55:49.556170 32149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:46753: Network error: Client connection negotiation failed: client connection to 127.9.93.60:46753: connect: Connection refused (error 111)
I20260928 16:55:49.570787 32232 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.60:46753
I20260928 16:55:49.570844 32300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.60:46753 every 8 connection(s)
I20260928 16:55:49.572435 32232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-2/data/info.pb
I20260928 16:55:49.576968 32301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:49.581359 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 32232
I20260928 16:55:49.581614 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790614166338986-9588-0/minicluster-data/master-2/wal/instance
I20260928 16:55:49.582216 9588 external_mini_cluster.cc:1428] Setting key 0290fa9a13e15535bbf0b91d4763ba6b
I20260928 16:55:49.586230 32301 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46549 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:55:49 dist-test-slave-wzcb krb5kdc[32068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.60@KRBTEST.COM
Sep 28 16:55:49 dist-test-slave-wzcb krb5kdc[32068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614549, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.60@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:55:49.609392 32305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.591053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42904 (local address 127.9.93.62:46549)
0928 16:55:49.591664 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:49.591671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:49.592222 (+ 551us) server_negotiation.cc:407] Connection header received
0928 16:55:49.592410 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:49.592415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:49.592482 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:49.592583 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:49.594169 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.595168 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.597196 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.597700 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.597841 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:49.602022 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:49.602032 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:49.602037 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:49.602098 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:49.605519 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:49.606477 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:49.606482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:49.606485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:49.606640 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:49.607099 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:49.607104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:49.607107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:49.607484 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:49.607688 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:49.608437 (+ 749us) server_negotiation.cc:299] Negotiation successful
0928 16:55:49.608722 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":325,"threads_started":1}
I20260928 16:55:49.610343 32304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.584854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59664 (local address 127.9.93.60:46753)
0928 16:55:49.585793 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:49.585808 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:49.585859 (+ 51us) server_negotiation.cc:407] Connection header received
0928 16:55:49.587685 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:49.587719 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:49.588299 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:49.588737 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:49.590771 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.592847 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.594639 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.595745 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.595896 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:49.601229 (+ 5333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:49.601240 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:49.601256 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:49.601305 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:49.605015 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:49.606043 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:49.606052 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:49.606054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:49.606271 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:49.606735 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:49.606741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:49.606744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:49.607689 (+ 945us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:49.608002 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:49.608489 (+ 487us) server_negotiation.cc:299] Negotiation successful
0928 16:55:49.608882 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":429,"threads_started":1}
I20260928 16:55:49.610365 32306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.591177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:46549 (local address 127.0.0.1:42904)
0928 16:55:49.592036 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.592099 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:49.592279 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:49.592674 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:49.592682 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:49.593614 (+ 932us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:49.593978 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:49.593992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.595456 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.595461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:49.596971 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:49.596989 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.598004 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.598012 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:49.598227 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:49.599082 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:55:49.599122 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:49.601778 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:49.605688 (+ 3910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:49.605705 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:49.605722 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:49.606298 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:49.606761 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:49.606774 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:49.606776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:49.606962 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:49.607655 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:49.607662 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:49.607950 (+ 288us) client_negotiation.cc:769] Sending connection context
0928 16:55:49.608284 (+ 334us) client_negotiation.cc:240] Negotiation successful
0928 16:55:49.609063 (+ 779us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":198,"thread_start_us":473,"threads_started":1}
I20260928 16:55:49.612286 32301 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:55:49 dist-test-slave-wzcb krb5kdc[32068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614549, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.60@KRBTEST.COM for kudu/127.9.93.61@KRBTEST.COM
Sep 28 16:55:49 dist-test-slave-wzcb krb5kdc[32068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:55:49.628671 32306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.613022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:37681 (local address 127.0.0.1:42952)
0928 16:55:49.613187 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.613209 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:49.613338 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:49.613694 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:49.613699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:49.614221 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:49.614466 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:49.614480 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.615927 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.615931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:49.617251 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:49.617270 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.618311 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.618316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:49.618518 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:49.619208 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:55:49.619235 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:49.621971 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:49.626022 (+ 4051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:49.626028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:49.626032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:49.626468 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:49.626957 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:49.626961 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:49.626964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:49.627125 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:49.627798 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:49.627802 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:49.627962 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 16:55:49.628080 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 16:55:49.628333 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 16:55:49.628775 32229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.613147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42952 (local address 127.9.93.61:37681)
0928 16:55:49.613338 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:49.613344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:49.613363 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:55:49.613484 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:49.613490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:49.613561 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:49.613647 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:49.614651 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.615728 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.617435 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.618077 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.618235 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:49.622260 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:49.622269 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:49.622272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:49.622319 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:49.625847 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:49.626608 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:49.626613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:49.626617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:49.626812 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:49.627238 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:49.627243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:49.627246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:49.627656 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:49.627836 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:49.628222 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 16:55:49.628477 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:55:49.629777 32301 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:55:49.631713 32305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.614327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42914 (local address 127.9.93.62:46549)
0928 16:55:49.614521 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:49.614527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:49.614546 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:55:49.614659 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:49.614662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:49.614735 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:49.614805 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:49.616363 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.617475 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.619177 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.619933 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.620084 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:49.625055 (+ 4971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:49.625062 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:49.625067 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:49.625117 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:49.628360 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:49.629363 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:49.629368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:49.629370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:49.629546 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:49.630021 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:49.630027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:49.630030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:49.630444 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:49.630916 (+ 472us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:49.631116 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 16:55:49.631395 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 16:55:49 dist-test-slave-wzcb krb5kdc[32068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614549, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.60@KRBTEST.COM for kudu/127.9.93.60@KRBTEST.COM
Sep 28 16:55:49 dist-test-slave-wzcb krb5kdc[32068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.61@KRBTEST.COM
I20260928 16:55:49.644374 32306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.630707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:46753 (local address 127.0.0.1:59672)
0928 16:55:49.630839 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.630858 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:49.630970 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:49.631347 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:49.631352 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:49.631835 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:49.632043 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:49.632053 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.633211 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.633216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:49.634311 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:49.634327 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.635004 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.635010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:49.635230 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:49.635898 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:55:49.635921 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:49.638249 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:49.641489 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:49.641496 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:49.641500 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:49.642056 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:49.642434 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:49.642438 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:49.642440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:49.642574 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:49.643215 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:49.643221 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:49.643424 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 16:55:49.643601 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 16:55:49.643977 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 16:55:49.644480 32304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.630649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.9.93.60:46753)
0928 16:55:49.630837 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:49.630843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:49.630914 (+ 71us) server_negotiation.cc:407] Connection header received
0928 16:55:49.631097 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:49.631103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:49.631182 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:49.631276 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:49.632178 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.633070 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.634433 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.634867 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.634981 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:49.638466 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:49.638473 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:49.638478 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:49.638521 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:49.641320 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:49.642179 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:49.642184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:49.642187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:49.642320 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:49.642684 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:49.642690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:49.642693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:49.643097 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:49.643217 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:49.643749 (+ 532us) server_negotiation.cc:299] Negotiation successful
0928 16:55:49.644127 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":150,"server-negotiator.queue_time_us":61}
I20260928 16:55:49.651083 32301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20673971547741f6aab0f8fdd9b28b6e: Bootstrap starting.
I20260928 16:55:49.651568 32229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.633748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42958 (local address 127.9.93.61:37681)
0928 16:55:49.633950 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:49.633955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:49.634166 (+ 211us) server_negotiation.cc:407] Connection header received
0928 16:55:49.634337 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:49.634341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:49.634409 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:49.634532 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:49.636036 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.636784 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.638393 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.639039 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.639191 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:49.644511 (+ 5320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:49.644520 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:49.644524 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:49.644576 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:49.648293 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:49.649243 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:49.649249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:49.649253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:49.649447 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:49.649832 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:49.649838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:49.649840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:49.650252 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:49.650547 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:49.650907 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 16:55:49.651175 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:55:49.655215 32301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20673971547741f6aab0f8fdd9b28b6e: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:49.656767 32301 log.cc:824] T 00000000000000000000000000000000 P 20673971547741f6aab0f8fdd9b28b6e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:49.657421 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:55:49.660702 32301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20673971547741f6aab0f8fdd9b28b6e: No bootstrap required, opened a new log
I20260928 16:55:49.661260 32124 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "513846aa62c8459980abc7da7584a959"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42914
I20260928 16:55:49.672353 32301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20673971547741f6aab0f8fdd9b28b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513846aa62c8459980abc7da7584a959" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46549 } } peers { permanent_uuid: "cff6e6ea6e6e48089b71aebe4cffbdc5" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } } peers { permanent_uuid: "20673971547741f6aab0f8fdd9b28b6e" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } }
I20260928 16:55:49.672880 32301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20673971547741f6aab0f8fdd9b28b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20673971547741f6aab0f8fdd9b28b6e, State: Initialized, Role: FOLLOWER
I20260928 16:55:49.673565 32301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20673971547741f6aab0f8fdd9b28b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513846aa62c8459980abc7da7584a959" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46549 } } peers { permanent_uuid: "cff6e6ea6e6e48089b71aebe4cffbdc5" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } } peers { permanent_uuid: "20673971547741f6aab0f8fdd9b28b6e" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } }
I20260928 16:55:49.674701 32308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20673971547741f6aab0f8fdd9b28b6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513846aa62c8459980abc7da7584a959" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46549 } } peers { permanent_uuid: "cff6e6ea6e6e48089b71aebe4cffbdc5" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } } peers { permanent_uuid: "20673971547741f6aab0f8fdd9b28b6e" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } } }
I20260928 16:55:49.675076 32308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20673971547741f6aab0f8fdd9b28b6e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:55:49.676173 32301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20673971547741f6aab0f8fdd9b28b6e [sys.catalog]: configured and running, proceeding with master startup.
W20260928 16:55:49.687456 32319 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 20673971547741f6aab0f8fdd9b28b6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:55:49.687704 32319 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:55:49.684566 32232 master_runner.cc:186] Error getting master registration for 127.9.93.62:46549: OK, instance_id { permanent_uuid: "513846aa62c8459980abc7da7584a959" instance_seqno: 1790614548405950 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:55:49.694294 32232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:55:49.709376 32224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } attempt: 5
Sep 28 16:55:49 dist-test-slave-wzcb krb5kdc[32068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614548, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.61@KRBTEST.COM for kudu/127.9.93.60@KRBTEST.COM
I20260928 16:55:49.728132 32304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.710418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59686 (local address 127.9.93.60:46753)
0928 16:55:49.710660 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:49.710666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:49.710690 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:55:49.710800 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:49.710806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:49.710898 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:49.710996 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:49.711951 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.712978 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.714918 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.715540 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.715677 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:49.720208 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:49.720216 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:49.720222 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:49.720287 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:49.724310 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:49.725429 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:49.725435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:49.725439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:49.725648 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:49.726169 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:49.726174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:49.726178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:49.726686 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:49.726864 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:49.727496 (+ 632us) server_negotiation.cc:299] Negotiation successful
0928 16:55:49.727743 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 16:55:49.728127 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.710284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:46753 (local address 127.0.0.1:59686)
0928 16:55:49.710489 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.710514 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:49.710662 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:49.711058 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:49.711063 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:49.711567 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:49.711764 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:49.711776 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.713177 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.713183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:49.714716 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:49.714738 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.715719 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.715725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:49.715945 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:49.716653 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:55:49.716678 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:49.719919 (+ 3241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:49.724552 (+ 4633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:49.724559 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:49.724564 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:49.725274 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:49.725799 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:49.725805 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:49.725809 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:49.726033 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:49.726955 (+ 922us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:49.726960 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:49.727153 (+ 193us) client_negotiation.cc:769] Sending connection context
0928 16:55:49.727348 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 16:55:49.727710 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 16:55:49.735627 32224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff6e6ea6e6e48089b71aebe4cffbdc5: Bootstrap starting.
I20260928 16:55:49.740770 32224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cff6e6ea6e6e48089b71aebe4cffbdc5: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:49.742477 32224 log.cc:824] T 00000000000000000000000000000000 P cff6e6ea6e6e48089b71aebe4cffbdc5: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:49.746692 32224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff6e6ea6e6e48089b71aebe4cffbdc5: No bootstrap required, opened a new log
I20260928 16:55:49.758652 32224 raft_consensus.cc:348] T 00000000000000000000000000000000 P cff6e6ea6e6e48089b71aebe4cffbdc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513846aa62c8459980abc7da7584a959" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46549 } } peers { permanent_uuid: "cff6e6ea6e6e48089b71aebe4cffbdc5" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } } peers { permanent_uuid: "20673971547741f6aab0f8fdd9b28b6e" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } }
I20260928 16:55:49.759162 32224 raft_consensus.cc:729] T 00000000000000000000000000000000 P cff6e6ea6e6e48089b71aebe4cffbdc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff6e6ea6e6e48089b71aebe4cffbdc5, State: Initialized, Role: FOLLOWER
I20260928 16:55:49.759859 32224 consensus_queue.cc:260] T 00000000000000000000000000000000 P cff6e6ea6e6e48089b71aebe4cffbdc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513846aa62c8459980abc7da7584a959" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46549 } } peers { permanent_uuid: "cff6e6ea6e6e48089b71aebe4cffbdc5" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } } peers { permanent_uuid: "20673971547741f6aab0f8fdd9b28b6e" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } }
I20260928 16:55:49.761126 32323 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff6e6ea6e6e48089b71aebe4cffbdc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513846aa62c8459980abc7da7584a959" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46549 } } peers { permanent_uuid: "cff6e6ea6e6e48089b71aebe4cffbdc5" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } } peers { permanent_uuid: "20673971547741f6aab0f8fdd9b28b6e" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } } }
I20260928 16:55:49.761600 32323 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff6e6ea6e6e48089b71aebe4cffbdc5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:55:49.762786 32224 sys_catalog.cc:564] T 00000000000000000000000000000000 P cff6e6ea6e6e48089b71aebe4cffbdc5 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 16:55:49.774183 32334 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cff6e6ea6e6e48089b71aebe4cffbdc5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:55:49.774374 32334 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:55:49.771205 32156 master_runner.cc:186] Error getting master registration for 127.9.93.62:46549: OK, instance_id { permanent_uuid: "513846aa62c8459980abc7da7584a959" instance_seqno: 1790614548405950 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:55:49.783963 32156 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 16:55:49.803354 32254 tablet.cc:2369] T 00000000000000000000000000000000 P 20673971547741f6aab0f8fdd9b28b6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:55:49.833411 32149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } attempt: 5
Sep 28 16:55:49 dist-test-slave-wzcb krb5kdc[32068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614548, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for kudu/127.9.93.60@KRBTEST.COM
I20260928 16:55:49.854094 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.834281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:46753 (local address 127.0.0.1:59694)
0928 16:55:49.834817 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:49.834840 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:49.834996 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:49.835445 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:49.835451 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:49.836013 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:49.836253 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:49.836268 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.837827 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.837834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:49.839376 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:49.839399 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.840896 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.840903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:49.841119 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:49.841901 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:55:49.841929 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:49.845255 (+ 3326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:49.850218 (+ 4963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:49.850225 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:49.850229 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:49.850868 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:49.851511 (+ 643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:49.851517 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:49.851521 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:49.851803 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:49.852873 (+ 1070us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:49.852880 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:49.853076 (+ 196us) client_negotiation.cc:769] Sending connection context
0928 16:55:49.853248 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 16:55:49.853631 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362}
I20260928 16:55:49.855206 32304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.834765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59694 (local address 127.9.93.60:46753)
0928 16:55:49.834976 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:49.834982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:49.835010 (+ 28us) server_negotiation.cc:407] Connection header received
0928 16:55:49.835120 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:49.835126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:49.835229 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:49.835410 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:49.836454 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.837656 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.839638 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.840445 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.840591 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:49.845639 (+ 5048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:49.845650 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:49.845659 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:49.845715 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:49.849951 (+ 4236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:49.851073 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:49.851081 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:49.851085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:49.851335 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:49.851970 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:49.851976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:49.851980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:49.852587 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:49.852767 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:49.854514 (+ 1747us) server_negotiation.cc:299] Negotiation successful
0928 16:55:49.854822 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:55:49.862875 32149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959: Bootstrap starting.
I20260928 16:55:49.868412 32149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:49.870461 32149 log.cc:824] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:49.875902 32149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959: No bootstrap required, opened a new log
I20260928 16:55:49.885152 32149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513846aa62c8459980abc7da7584a959" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46549 } } peers { permanent_uuid: "cff6e6ea6e6e48089b71aebe4cffbdc5" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } } peers { permanent_uuid: "20673971547741f6aab0f8fdd9b28b6e" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } }
I20260928 16:55:49.885784 32149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513846aa62c8459980abc7da7584a959, State: Initialized, Role: FOLLOWER
I20260928 16:55:49.886504 32149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513846aa62c8459980abc7da7584a959" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46549 } } peers { permanent_uuid: "cff6e6ea6e6e48089b71aebe4cffbdc5" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } } peers { permanent_uuid: "20673971547741f6aab0f8fdd9b28b6e" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } }
I20260928 16:55:49.888136 32337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513846aa62c8459980abc7da7584a959" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46549 } } peers { permanent_uuid: "cff6e6ea6e6e48089b71aebe4cffbdc5" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } } peers { permanent_uuid: "20673971547741f6aab0f8fdd9b28b6e" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } } }
I20260928 16:55:49.888983 32337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:55:49.889815 32149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:55:49.890300 32124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 16:55:49.890534 32124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:49.894238 32124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513846aa62c8459980abc7da7584a959" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46549 } } peers { permanent_uuid: "cff6e6ea6e6e48089b71aebe4cffbdc5" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } } peers { permanent_uuid: "20673971547741f6aab0f8fdd9b28b6e" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } }
W20260928 16:55:49.895606 32103 tablet.cc:2369] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:55:49.897550 32198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "513846aa62c8459980abc7da7584a959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cff6e6ea6e6e48089b71aebe4cffbdc5"
I20260928 16:55:49.898150 32198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cff6e6ea6e6e48089b71aebe4cffbdc5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:49.902143 32198 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cff6e6ea6e6e48089b71aebe4cffbdc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 513846aa62c8459980abc7da7584a959 in term 1.
I20260928 16:55:49.903214 32100 leader_election.cc:302] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 513846aa62c8459980abc7da7584a959, cff6e6ea6e6e48089b71aebe4cffbdc5; no voters:
I20260928 16:55:49.905289 32276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "513846aa62c8459980abc7da7584a959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "20673971547741f6aab0f8fdd9b28b6e"
I20260928 16:55:49.905930 32276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20673971547741f6aab0f8fdd9b28b6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:49.910109 32276 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 20673971547741f6aab0f8fdd9b28b6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 513846aa62c8459980abc7da7584a959 in term 1.
I20260928 16:55:49.911130 32124 leader_election.cc:288] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959 [CANDIDATE]: Term 1 election: Requested vote from peers cff6e6ea6e6e48089b71aebe4cffbdc5 (127.9.93.61:37681), 20673971547741f6aab0f8fdd9b28b6e (127.9.93.60:46753)
I20260928 16:55:49.912521 32337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:55:49.920099 32337 raft_consensus.cc:686] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959 [term 1 LEADER]: Becoming Leader. State: Replica: 513846aa62c8459980abc7da7584a959, State: Running, Role: LEADER
I20260928 16:55:49.921133 32337 consensus_queue.cc:237] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513846aa62c8459980abc7da7584a959" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46549 } } peers { permanent_uuid: "cff6e6ea6e6e48089b71aebe4cffbdc5" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } } peers { permanent_uuid: "20673971547741f6aab0f8fdd9b28b6e" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } }
I20260928 16:55:49.933903 32341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 513846aa62c8459980abc7da7584a959. Latest consensus state: current_term: 1 leader_uuid: "513846aa62c8459980abc7da7584a959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513846aa62c8459980abc7da7584a959" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46549 } } peers { permanent_uuid: "cff6e6ea6e6e48089b71aebe4cffbdc5" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } } peers { permanent_uuid: "20673971547741f6aab0f8fdd9b28b6e" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } } }
I20260928 16:55:49.934613 32341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959 [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:49.935791 32348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:55:49.956117 32348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260928 16:55:49.957958 32177 tablet.cc:2369] T 00000000000000000000000000000000 P cff6e6ea6e6e48089b71aebe4cffbdc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:55:49.975102 32360 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 16:55:49.988998 32276 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 20673971547741f6aab0f8fdd9b28b6e [term 1 FOLLOWER]: Refusing update from remote peer 513846aa62c8459980abc7da7584a959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:55:49.990696 32341 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20673971547741f6aab0f8fdd9b28b6e" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:55:49.994290 32198 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cff6e6ea6e6e48089b71aebe4cffbdc5 [term 1 FOLLOWER]: Refusing update from remote peer 513846aa62c8459980abc7da7584a959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:55:49.995664 32341 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff6e6ea6e6e48089b71aebe4cffbdc5" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:55:50.012742 32305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.955462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42930 (local address 127.9.93.62:46549)
0928 16:55:49.955680 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:49.955687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:49.965815 (+ 10128us) server_negotiation.cc:407] Connection header received
0928 16:55:49.967224 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:49.967230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:49.967353 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:49.967439 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:49.975801 (+ 8362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.977020 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.982157 (+ 5137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.982888 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.982996 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:50.001559 (+ 18563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:50.001568 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:50.001574 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:50.001652 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:50.005581 (+ 3929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:50.007445 (+ 1864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:50.007452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:50.007456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:50.007641 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:50.008998 (+ 1357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:50.009005 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:50.009009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:50.009471 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:50.009815 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:50.011840 (+ 2025us) server_negotiation.cc:299] Negotiation successful
0928 16:55:50.012261 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:55:50.014817 32304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.951829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59704 (local address 127.9.93.60:46753)
0928 16:55:49.952061 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:49.952067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:49.965171 (+ 13104us) server_negotiation.cc:407] Connection header received
0928 16:55:49.970085 (+ 4914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:49.970094 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:49.970218 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:49.970318 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:49.979039 (+ 8721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.980136 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.991483 (+ 11347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.992311 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.992461 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:49.999229 (+ 6768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:49.999238 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:49.999243 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:49.999295 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:50.003461 (+ 4166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:50.008200 (+ 4739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:50.008212 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:50.008216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:50.008440 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:50.010374 (+ 1934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:50.010380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:50.010384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:50.010842 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:50.011176 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:50.014059 (+ 2883us) server_negotiation.cc:299] Negotiation successful
0928 16:55:50.014391 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 16:55:50.032951 32323 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff6e6ea6e6e48089b71aebe4cffbdc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 513846aa62c8459980abc7da7584a959. Latest consensus state: current_term: 1 leader_uuid: "513846aa62c8459980abc7da7584a959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513846aa62c8459980abc7da7584a959" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46549 } } peers { permanent_uuid: "cff6e6ea6e6e48089b71aebe4cffbdc5" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } } peers { permanent_uuid: "20673971547741f6aab0f8fdd9b28b6e" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } } }
I20260928 16:55:50.033398 32323 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff6e6ea6e6e48089b71aebe4cffbdc5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:55:50.047436 32359 mvcc.cc:205] Tried to move back new op lower bound from 7334357196738826240 to 7334357196484898816. Current Snapshot: MvccSnapshot[applied={T|T < 7334357196738826240}]
I20260928 16:55:50.048435 32308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20673971547741f6aab0f8fdd9b28b6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 513846aa62c8459980abc7da7584a959. Latest consensus state: current_term: 1 leader_uuid: "513846aa62c8459980abc7da7584a959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513846aa62c8459980abc7da7584a959" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46549 } } peers { permanent_uuid: "cff6e6ea6e6e48089b71aebe4cffbdc5" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } } peers { permanent_uuid: "20673971547741f6aab0f8fdd9b28b6e" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } } }
I20260928 16:55:50.048847 32308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20673971547741f6aab0f8fdd9b28b6e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:55:50.055068 32361 mvcc.cc:205] Tried to move back new op lower bound from 7334357196738826240 to 7334357196484898816. Current Snapshot: MvccSnapshot[applied={T|T < 7334357196738826240}]
I20260928 16:55:50.065043 32337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "513846aa62c8459980abc7da7584a959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513846aa62c8459980abc7da7584a959" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46549 } } peers { permanent_uuid: "cff6e6ea6e6e48089b71aebe4cffbdc5" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } } peers { permanent_uuid: "20673971547741f6aab0f8fdd9b28b6e" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } } }
I20260928 16:55:50.065467 32337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959 [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:50.065815 32337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "513846aa62c8459980abc7da7584a959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513846aa62c8459980abc7da7584a959" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46549 } } peers { permanent_uuid: "cff6e6ea6e6e48089b71aebe4cffbdc5" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } } peers { permanent_uuid: "20673971547741f6aab0f8fdd9b28b6e" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } } }
I20260928 16:55:50.066131 32337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959 [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:50.065982 32323 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff6e6ea6e6e48089b71aebe4cffbdc5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "513846aa62c8459980abc7da7584a959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513846aa62c8459980abc7da7584a959" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46549 } } peers { permanent_uuid: "cff6e6ea6e6e48089b71aebe4cffbdc5" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } } peers { permanent_uuid: "20673971547741f6aab0f8fdd9b28b6e" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } } }
I20260928 16:55:50.066249 32323 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff6e6ea6e6e48089b71aebe4cffbdc5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:55:50.073110 32348 catalog_manager.cc:1347] Generated new cluster ID: 8c02f57913bd480faffbb5a99a461ef1
I20260928 16:55:50.073267 32348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:55:50.074550 32229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:49.958803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42968 (local address 127.9.93.61:37681)
0928 16:55:49.958998 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:49.959004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:49.966640 (+ 7636us) server_negotiation.cc:407] Connection header received
0928 16:55:49.966875 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:49.966882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:49.967010 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:49.967120 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:49.969869 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.971036 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.973237 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:49.973981 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:49.974091 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:50.034161 (+ 60070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:50.034169 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:50.034175 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:50.034249 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:50.037948 (+ 3699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:50.044932 (+ 6984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:50.044940 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:50.044943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:50.045180 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:50.045789 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:50.045795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:50.045797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:50.046272 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:50.046537 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:50.073768 (+ 27231us) server_negotiation.cc:299] Negotiation successful
0928 16:55:50.074143 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 16:55:50.091647 32308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20673971547741f6aab0f8fdd9b28b6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "513846aa62c8459980abc7da7584a959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513846aa62c8459980abc7da7584a959" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 46549 } } peers { permanent_uuid: "cff6e6ea6e6e48089b71aebe4cffbdc5" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 37681 } } peers { permanent_uuid: "20673971547741f6aab0f8fdd9b28b6e" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 46753 } } }
I20260928 16:55:50.091989 32308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20673971547741f6aab0f8fdd9b28b6e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:55:50.097550 32348 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:55:50.099701 32348 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:55:50.125715 32348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 513846aa62c8459980abc7da7584a959: Generated new TSK 0
I20260928 16:55:50.127123 32348 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:55:50.291850 32305 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:55:50.266181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42938 (local address 127.9.93.62:46549)
0928 16:55:50.266410 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:50.266418 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:50.288495 (+ 22077us) server_negotiation.cc:407] Connection header received
0928 16:55:50.289188 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:50.289196 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:50.289315 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:50.289416 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:50.291551 (+ 2135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42938: BlockingRecv error: recv got EOF from 127.0.0.1:42938 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260928 16:55:50.296804 32304 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:55:50.273139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59720 (local address 127.9.93.60:46753)
0928 16:55:50.273324 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:50.273329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:50.291011 (+ 17682us) server_negotiation.cc:407] Connection header received
0928 16:55:50.293374 (+ 2363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:50.293382 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:50.293518 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:50.293608 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:50.296524 (+ 2916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59720: BlockingRecv error: recv got EOF from 127.0.0.1:59720 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 16:55:50.298271 32229 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:55:50.272719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42980 (local address 127.9.93.61:37681)
0928 16:55:50.272934 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:50.272941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:50.292767 (+ 19826us) server_negotiation.cc:407] Connection header received
0928 16:55:50.294154 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:50.294161 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:50.294301 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:50.294387 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:50.297999 (+ 3612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42980: BlockingRecv error: recv got EOF from 127.0.0.1:42980 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260928 16:55:50.309197 32229 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:55:50.306387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42984 (local address 127.9.93.61:37681)
0928 16:55:50.306625 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:50.306632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:50.307282 (+ 650us) server_negotiation.cc:407] Connection header received
0928 16:55:50.307629 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:50.307636 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:50.307785 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:50.307918 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:50.308894 (+ 976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42984: BlockingRecv error: recv got EOF from 127.0.0.1:42984 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260928 16:55:50.317656 32229 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:55:50.314457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42998 (local address 127.9.93.61:37681)
0928 16:55:50.314685 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:50.314693 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:50.315910 (+ 1217us) server_negotiation.cc:407] Connection header received
0928 16:55:50.316153 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:50.316159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:50.316269 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:50.316380 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:50.317412 (+ 1032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42998: BlockingRecv error: recv got EOF from 127.0.0.1:42998 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 16:55:50.329830 32229 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:55:50.324741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43002 (local address 127.9.93.61:37681)
0928 16:55:50.324940 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:50.324947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:50.328038 (+ 3091us) server_negotiation.cc:407] Connection header received
0928 16:55:50.328307 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:50.328314 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:50.328443 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:50.328550 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:50.329532 (+ 982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43002: BlockingRecv error: recv got EOF from 127.0.0.1:43002 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 16:55:50.350131 32229 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:55:50.341870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43018 (local address 127.9.93.61:37681)
0928 16:55:50.342111 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:50.342118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:50.348254 (+ 6136us) server_negotiation.cc:407] Connection header received
0928 16:55:50.348501 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:50.348507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:50.348652 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:50.348781 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:50.349843 (+ 1062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43018: BlockingRecv error: recv got EOF from 127.0.0.1:43018 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260928 16:55:50.367983 32229 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:55:50.365236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43024 (local address 127.9.93.61:37681)
0928 16:55:50.365491 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:50.365497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:50.366147 (+ 650us) server_negotiation.cc:407] Connection header received
0928 16:55:50.366423 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:50.366429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:50.366534 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:50.366656 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:50.367700 (+ 1044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43024: BlockingRecv error: recv got EOF from 127.0.0.1:43024 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260928 16:55:50.396030 32229 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:55:50.393303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43040 (local address 127.9.93.61:37681)
0928 16:55:50.393526 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:50.393533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:50.394196 (+ 663us) server_negotiation.cc:407] Connection header received
0928 16:55:50.394474 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:50.394481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:50.394587 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:50.394701 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:50.395771 (+ 1070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43040: BlockingRecv error: recv got EOF from 127.0.0.1:43040 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260928 16:55:50.443734 32229 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:55:50.441303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43048 (local address 127.9.93.61:37681)
0928 16:55:50.441551 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:50.441558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:50.441973 (+ 415us) server_negotiation.cc:407] Connection header received
0928 16:55:50.442195 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:50.442200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:50.442285 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:50.442388 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:50.443443 (+ 1055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43048: BlockingRecv error: recv got EOF from 127.0.0.1:43048 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260928 16:55:50.514210 32229 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:55:50.511978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.9.93.61:37681)
0928 16:55:50.512237 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:50.512245 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:50.512651 (+ 406us) server_negotiation.cc:407] Connection header received
0928 16:55:50.512862 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:50.512868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:50.512980 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:50.513114 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:50.513956 (+ 842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43056: BlockingRecv error: recv got EOF from 127.0.0.1:43056 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 16:55:50.648494 32229 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:55:50.646212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43060 (local address 127.9.93.61:37681)
0928 16:55:50.646514 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:50.646522 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:50.646778 (+ 256us) server_negotiation.cc:407] Connection header received
0928 16:55:50.647041 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:50.647046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:50.647131 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:50.647226 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:50.648212 (+ 986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43060: BlockingRecv error: recv got EOF from 127.0.0.1:43060 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
I20260928 16:55:50.690156 32319 catalog_manager.cc:1259] Loaded cluster ID: 8c02f57913bd480faffbb5a99a461ef1
I20260928 16:55:50.690300 32319 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 20673971547741f6aab0f8fdd9b28b6e: loading cluster ID for follower catalog manager: success
I20260928 16:55:50.693459 32319 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 20673971547741f6aab0f8fdd9b28b6e: acquiring CA information for follower catalog manager: success
I20260928 16:55:50.695788 32319 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 20673971547741f6aab0f8fdd9b28b6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:55:50.776885 32334 catalog_manager.cc:1259] Loaded cluster ID: 8c02f57913bd480faffbb5a99a461ef1
I20260928 16:55:50.777011 32334 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cff6e6ea6e6e48089b71aebe4cffbdc5: loading cluster ID for follower catalog manager: success
I20260928 16:55:50.780004 32334 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cff6e6ea6e6e48089b71aebe4cffbdc5: acquiring CA information for follower catalog manager: success
I20260928 16:55:50.782142 32334 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cff6e6ea6e6e48089b71aebe4cffbdc5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:55:50.919361 32229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:50.912852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43068 (local address 127.9.93.61:37681)
0928 16:55:50.913115 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:50.913121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:50.913379 (+ 258us) server_negotiation.cc:407] Connection header received
0928 16:55:50.913634 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:50.913638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:50.913826 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:50.913941 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:55:50.914480 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:50.915245 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:50.916835 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:50.917280 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:50.917404 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:50.918029 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:55:50.918635 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:55:50.918799 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:50.918908 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 16:55:50.919012 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260928 16:55:50.933123 32432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:50.926641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59734 (local address 127.9.93.60:46753)
0928 16:55:50.927183 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:50.927188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:50.927245 (+ 57us) server_negotiation.cc:407] Connection header received
0928 16:55:50.927535 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:50.927539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:50.927732 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:50.927904 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:55:50.928340 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:50.929068 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:50.930675 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:50.931116 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:50.931221 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:50.931748 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:55:50.932475 (+ 727us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:55:50.932611 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:50.932725 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 16:55:50.932817 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":313,"threads_started":1}
I20260928 16:55:50.948670 32438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:50.940043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42944 (local address 127.9.93.62:46549)
0928 16:55:50.942024 (+ 1981us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:50.942031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:50.942054 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:55:50.942205 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:50.942211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:50.942489 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:50.942702 (+ 213us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:55:50.943441 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:50.944473 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:50.946181 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:50.946604 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:50.946723 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:50.947379 (+ 656us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:55:50.947930 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:55:50.948030 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:50.948238 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 16:55:50.948345 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1834,"thread_start_us":1705,"threads_started":1}
I20260928 16:55:50.949272 32432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:50.940671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59736 (local address 127.9.93.60:46753)
0928 16:55:50.941632 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:50.941638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:50.942416 (+ 778us) server_negotiation.cc:407] Connection header received
0928 16:55:50.942592 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:50.942598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:50.942682 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:50.942772 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:55:50.943718 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:50.944800 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:50.947167 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:50.947808 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:50.947940 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:50.948461 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:55:50.948615 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:55:50.948681 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:50.948912 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 16:55:50.949008 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":810}
I20260928 16:55:50.950196 32229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:50.940138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.9.93.61:37681)
0928 16:55:50.940713 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:50.940720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:50.943550 (+ 2830us) server_negotiation.cc:407] Connection header received
0928 16:55:50.943817 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:50.943823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:50.943916 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:50.944009 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:55:50.944601 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:50.945614 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:50.947704 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:50.948296 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:50.948412 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:50.949408 (+ 996us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:55:50.949579 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:55:50.949697 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:50.949873 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 16:55:50.949967 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422}
I20260928 16:55:50.955369 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 32081
I20260928 16:55:50.956629 32144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:55:51.072490 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 32081
I20260928 16:55:51.093833 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 32156
I20260928 16:55:51.095091 32219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:55:51.211647 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 32156
I20260928 16:55:51.230830 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 32232
I20260928 16:55:51.232100 32296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:55:51.342268 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 32232
2026-09-28T16:55:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4823 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:55:51 dist-test-slave-wzcb krb5kdc[32446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:55:51 dist-test-slave-wzcb krb5kdc[32446](info): set up 2 sockets
Sep 28 16:55:51 dist-test-slave-wzcb krb5kdc[32446](info): commencing operation
krb5kdc: starting...
W20260928 16:55:52.402372 9588 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.005s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/krb5kdc/test-user.keytab.
Sep 28 16:55:52 dist-test-slave-wzcb krb5kdc[32446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:55:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:55:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
Entry for principal kudu/127.9.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab.
I20260928 16:55:52.740314 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:39481
--webserver_interface=127.9.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:39481,127.9.93.61:42991,127.9.93.60:41835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:55:53.121657 32459 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:55:53.122063 32459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:53.122174 32459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:53.122300 32459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:53.130365 32459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:53.130481 32459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:53.130545 32459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:53.130622 32459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:53.130718 32459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:53.141618 32459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:39481,127.9.93.61:42991,127.9.93.60:41835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.62.keytab
--principal=kudu/127.9.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.62:39481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:53.143146 32459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:53.144696 32459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:53.158565 32464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:53.158605 32465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:53.158638 32467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:53.159554 32459 server_base.cc:1034] running on GCE node
I20260928 16:55:53.160533 32459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:53.162340 32459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:53.163626 32459 hybrid_clock.cc:648] HybridClock initialized: now 1790614553163533 us; error 52 us; skew 500 ppm
I20260928 16:55:53.164088 32459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:53 dist-test-slave-wzcb krb5kdc[32446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614553, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:53.168005 32459 init.cc:377] Logged in from keytab as kudu/127.9.93.62@KRBTEST.COM (short username kudu)
I20260928 16:55:53.169860 32459 webserver.cc:460] Webserver started at http://127.9.93.62:44659/ using document root <none> and password file <none>
I20260928 16:55:53.170689 32459 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:53.170823 32459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:53.171242 32459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:53.174810 32459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-0/data/instance:
uuid: "19952321790f40119b0dc29ac875fb02"
format_stamp: "Formatted at 2026-09-28 16:55:53 on dist-test-slave-wzcb"
server_key: "faf44d64b00dc218e96a4f3bfab0b618"
server_key_iv: "5a3ef3a789e100c3baae6fd57486b683"
server_key_version: "encryptionkey@0"
I20260928 16:55:53.175765 32459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-0/wal/instance:
uuid: "19952321790f40119b0dc29ac875fb02"
format_stamp: "Formatted at 2026-09-28 16:55:53 on dist-test-slave-wzcb"
server_key: "faf44d64b00dc218e96a4f3bfab0b618"
server_key_iv: "5a3ef3a789e100c3baae6fd57486b683"
server_key_version: "encryptionkey@0"
I20260928 16:55:53.182077 32459 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:55:53.186390 32474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:53.188292 32459 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 16:55:53.188516 32459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-0/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-0/wal
uuid: "19952321790f40119b0dc29ac875fb02"
format_stamp: "Formatted at 2026-09-28 16:55:53 on dist-test-slave-wzcb"
server_key: "faf44d64b00dc218e96a4f3bfab0b618"
server_key_iv: "5a3ef3a789e100c3baae6fd57486b683"
server_key_version: "encryptionkey@0"
I20260928 16:55:53.188794 32459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:53.212456 32459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:53.217662 32459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:53.217991 32459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:53.238195 32459 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.62:39481
I20260928 16:55:53.238255 32525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.62:39481 every 8 connection(s)
I20260928 16:55:53.239842 32459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-0/data/info.pb
I20260928 16:55:53.243958 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 32459
I20260928 16:55:53.244203 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-0/wal/instance
I20260928 16:55:53.244777 32526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:53.244833 9588 external_mini_cluster.cc:1428] Setting key d0de674e9a27e832c3406511d09a9c32
I20260928 16:55:53.254920 32526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39481 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:55:53 dist-test-slave-wzcb krb5kdc[32446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614553, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.93.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.61@KRBTEST.COM" created.
I20260928 16:55:53.281834 32530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.259362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44172 (local address 127.9.93.62:39481)
0928 16:55:53.261386 (+ 2024us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:53.261396 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:53.261460 (+ 64us) server_negotiation.cc:407] Connection header received
0928 16:55:53.263172 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:53.263179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:53.263536 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:53.263739 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:53.265219 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:53.266523 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:53.268092 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:53.268711 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:53.268936 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:53.273421 (+ 4485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:53.273432 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:53.273450 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:53.273488 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:53.276770 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:53.277562 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:53.277570 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:53.277572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:53.277743 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:53.278168 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:53.278171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:53.278173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:53.278812 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:53.279025 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:53.279679 (+ 654us) server_negotiation.cc:299] Negotiation successful
0928 16:55:53.280119 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":1758,"thread_start_us":1499,"threads_started":1}
I20260928 16:55:53.281857 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.259414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:39481 (local address 127.0.0.1:44172)
0928 16:55:53.260427 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:53.260489 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:53.262733 (+ 2244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:53.263787 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:53.263794 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:53.264604 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:53.264990 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:53.265015 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:53.266693 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:53.266697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:53.267945 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:53.267959 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:53.268972 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:53.268979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:53.269147 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:53.270018 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:55:53.270048 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:53.272996 (+ 2948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:53.276934 (+ 3938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:53.276949 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:53.276963 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:53.277404 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:53.277864 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:53.277870 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:53.277874 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:53.278055 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:53.278951 (+ 896us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:53.278960 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:53.279215 (+ 255us) client_negotiation.cc:769] Sending connection context
0928 16:55:53.279513 (+ 298us) client_negotiation.cc:240] Negotiation successful
0928 16:55:53.280249 (+ 736us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"mutex_wait_us":172,"thread_start_us":435,"threads_started":1}
I20260928 16:55:53.285320 32526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:55:53.287086 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.286251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:42991 (local address 127.0.0.1:49496)
0928 16:55:53.286689 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:53.286873 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:42991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260928 16:55:53.287560 32477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.93.61:42991: connect: Connection refused (error 111)
I20260928 16:55:53.289599 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.289128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:42991 (local address 127.0.0.1:49508)
0928 16:55:53.289301 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:53.289403 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:42991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 16:55:53.289978 32526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.61:42991: Network error: Client connection negotiation failed: client connection to 127.9.93.61:42991: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.61.keytab.
Entry for principal kudu/127.9.93.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.61.keytab.
I20260928 16:55:53.317041 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.61:42991
--webserver_interface=127.9.93.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.61.keytab
--principal=kudu/127.9.93.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:39481,127.9.93.61:42991,127.9.93.60:41835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:55:53.339478 32526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } attempt: 1
I20260928 16:55:53.340953 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.340370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:42991 (local address 127.0.0.1:49514)
0928 16:55:53.340635 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:53.340771 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:42991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 16:55:53.342252 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.341810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:42991 (local address 127.0.0.1:49530)
0928 16:55:53.341982 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:53.342073 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:42991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 16:55:53.342491 32526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.61:42991: Network error: Client connection negotiation failed: client connection to 127.9.93.61:42991: connect: Connection refused (error 111)
I20260928 16:55:53.410806 32526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } attempt: 2
I20260928 16:55:53.412144 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.411614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:42991 (local address 127.0.0.1:49544)
0928 16:55:53.411887 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:53.411994 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:42991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 16:55:53.413463 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.412950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:42991 (local address 127.0.0.1:49554)
0928 16:55:53.413155 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:53.413281 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:42991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260928 16:55:53.413717 32526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.61:42991: Network error: Client connection negotiation failed: client connection to 127.9.93.61:42991: connect: Connection refused (error 111)
I20260928 16:55:53.505015 32526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } attempt: 3
I20260928 16:55:53.506347 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.505851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:42991 (local address 127.0.0.1:49570)
0928 16:55:53.506072 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:53.506176 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:42991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 16:55:53.507884 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.507386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:42991 (local address 127.0.0.1:49584)
0928 16:55:53.507583 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:53.507683 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:42991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260928 16:55:53.508160 32526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.61:42991: Network error: Client connection negotiation failed: client connection to 127.9.93.61:42991: connect: Connection refused (error 111)
I20260928 16:55:53.651475 32526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } attempt: 4
I20260928 16:55:53.652809 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.652196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:42991 (local address 127.0.0.1:49586)
0928 16:55:53.652552 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:53.652657 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:42991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199}
I20260928 16:55:53.654083 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.653658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:42991 (local address 127.0.0.1:49602)
0928 16:55:53.653804 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:53.653931 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.61:42991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 16:55:53.654338 32526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.61:42991: Network error: Client connection negotiation failed: client connection to 127.9.93.61:42991: connect: Connection refused (error 111)
W20260928 16:55:53.697182 32533 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:55:53.697599 32533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:53.697708 32533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:53.697804 32533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:53.705708 32533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:53.705839 32533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:53.705904 32533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:53.705957 32533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:53.706022 32533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:53.716894 32533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:39481,127.9.93.61:42991,127.9.93.60:41835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.61.keytab
--principal=kudu/127.9.93.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.61:42991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.93.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:53.718572 32533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:53.720234 32533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:53.734074 32539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:53.734140 32542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:53.734108 32540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:53.734917 32533 server_base.cc:1034] running on GCE node
I20260928 16:55:53.735993 32533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:53.737864 32533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:53.739233 32533 hybrid_clock.cc:648] HybridClock initialized: now 1790614553739011 us; error 122 us; skew 500 ppm
I20260928 16:55:53.739694 32533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:53 dist-test-slave-wzcb krb5kdc[32446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614553, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:53.743544 32533 init.cc:377] Logged in from keytab as kudu/127.9.93.61@KRBTEST.COM (short username kudu)
I20260928 16:55:53.745457 32533 webserver.cc:460] Webserver started at http://127.9.93.61:44035/ using document root <none> and password file <none>
I20260928 16:55:53.746338 32533 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:53.746455 32533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:53.746794 32533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:53.750577 32533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-1/data/instance:
uuid: "4df781a8a68547038f3688651fbaa9ae"
format_stamp: "Formatted at 2026-09-28 16:55:53 on dist-test-slave-wzcb"
server_key: "33a874da329f78ef432525c5838b6483"
server_key_iv: "859de00bcb478068f762a6346cec95f1"
server_key_version: "encryptionkey@0"
I20260928 16:55:53.751772 32533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-1/wal/instance:
uuid: "4df781a8a68547038f3688651fbaa9ae"
format_stamp: "Formatted at 2026-09-28 16:55:53 on dist-test-slave-wzcb"
server_key: "33a874da329f78ef432525c5838b6483"
server_key_iv: "859de00bcb478068f762a6346cec95f1"
server_key_version: "encryptionkey@0"
I20260928 16:55:53.758399 32533 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 16:55:53.762686 32549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:53.764586 32533 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:55:53.764819 32533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-1/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-1/wal
uuid: "4df781a8a68547038f3688651fbaa9ae"
format_stamp: "Formatted at 2026-09-28 16:55:53 on dist-test-slave-wzcb"
server_key: "33a874da329f78ef432525c5838b6483"
server_key_iv: "859de00bcb478068f762a6346cec95f1"
server_key_version: "encryptionkey@0"
I20260928 16:55:53.765069 32533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:53.782655 32533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:53.787642 32533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:53.787906 32533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:53.807714 32533 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.61:42991
I20260928 16:55:53.807729 32600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.61:42991 every 8 connection(s)
I20260928 16:55:53.809283 32533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-1/data/info.pb
I20260928 16:55:53.810153 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 32533
I20260928 16:55:53.810452 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-1/wal/instance
I20260928 16:55:53.810945 9588 external_mini_cluster.cc:1428] Setting key 19825ef018b552c5690f0fefa9a14ea9
I20260928 16:55:53.814036 32601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:53.822129 32601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39481 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.93.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.93.60@KRBTEST.COM" created.
Sep 28 16:55:53 dist-test-slave-wzcb krb5kdc[32446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614553, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.61@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:55:53.849794 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.825458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44184 (local address 127.9.93.62:39481)
0928 16:55:53.826116 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:53.826124 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:53.827371 (+ 1247us) server_negotiation.cc:407] Connection header received
0928 16:55:53.829833 (+ 2462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:53.829841 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:53.829981 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:53.830141 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:53.832283 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:53.833435 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:53.835437 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:53.836158 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:53.836339 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:53.842966 (+ 6627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:53.842973 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:53.842976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:53.843029 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:53.845971 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:53.846973 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:53.846978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:53.846981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:53.847142 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:53.847728 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:53.847733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:53.847735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:53.848064 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:53.848219 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:53.849111 (+ 892us) server_negotiation.cc:299] Negotiation successful
0928 16:55:53.849389 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":336,"threads_started":1}
I20260928 16:55:53.850996 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.826108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:39481 (local address 127.0.0.1:44184)
0928 16:55:53.827115 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:53.827186 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:53.829437 (+ 2251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:53.830443 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:53.830454 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:53.831155 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:53.831965 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:53.831998 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:53.833706 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:53.833711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:53.835198 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:53.835217 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:53.836479 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:53.836485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:53.836771 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:53.837571 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:55:53.837602 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:53.842591 (+ 4989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:53.846193 (+ 3602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:53.846204 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:53.846222 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:53.846787 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:53.847277 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:53.847283 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:53.847287 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:53.847581 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:53.848217 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:53.848227 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:53.848682 (+ 455us) client_negotiation.cc:769] Sending connection context
0928 16:55:53.848973 (+ 291us) client_negotiation.cc:240] Negotiation successful
0928 16:55:53.849616 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":379,"threads_started":1}
I20260928 16:55:53.853127 32601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:55:53 dist-test-slave-wzcb krb5kdc[32446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614553, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.61@KRBTEST.COM for kudu/127.9.93.61@KRBTEST.COM
I20260928 16:55:53.868695 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.854001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:42991 (local address 127.0.0.1:49614)
0928 16:55:53.854260 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:53.854282 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:53.854419 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:53.855597 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:53.855602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:53.856091 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:53.856239 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:53.856247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:53.857691 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:53.857695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:53.858542 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:53.858554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:53.859427 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:53.859433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:53.859631 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:53.860221 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:55:53.860238 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:53.862620 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:53.866006 (+ 3386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:53.866012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:53.866017 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:53.866423 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:53.866818 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:53.866823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:53.866827 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:53.866965 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:53.867648 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:53.867653 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:53.867808 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 16:55:53.867931 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 16:55:53.868262 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 16:55:53.868758 32608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.854266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49614 (local address 127.9.93.61:42991)
0928 16:55:53.855093 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:53.855099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:53.855127 (+ 28us) server_negotiation.cc:407] Connection header received
0928 16:55:53.855180 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:53.855186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:53.855444 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:53.855599 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:53.856411 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:53.857537 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:53.858674 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:53.859268 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:53.859418 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:53.862861 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:53.862877 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:53.862894 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:53.862927 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:53.865851 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:53.866570 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:53.866578 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:53.866580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:53.866700 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:53.867064 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:53.867068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:53.867069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:53.867513 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:53.867716 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:53.868140 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 16:55:53.868378 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":664,"thread_start_us":491,"threads_started":1}
I20260928 16:55:53.871978 32601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:55:53.873414 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.872802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41835 (local address 127.0.0.1:40704)
0928 16:55:53.873002 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:53.873195 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260928 16:55:53.873725 32552 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
I20260928 16:55:53.875591 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.875091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41835 (local address 127.0.0.1:40714)
0928 16:55:53.875281 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:53.875420 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 16:55:53.875869 32601 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:41835: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.93.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.60.keytab.
Entry for principal kudu/127.9.93.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.60.keytab.
I20260928 16:55:53.878890 9588 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkzF10S/build/asan/bin/kudu
/tmp/dist-test-taskkzF10S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.60:41835
--webserver_interface=127.9.93.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.60.keytab
--principal=kudu/127.9.93.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.93.20:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.93.62:39481,127.9.93.61:42991,127.9.93.60:41835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:55:53.925266 32601 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } attempt: 1
I20260928 16:55:53.926673 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.926124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41835 (local address 127.0.0.1:40720)
0928 16:55:53.926392 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:53.926499 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260928 16:55:53.927939 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.927569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41835 (local address 127.0.0.1:40732)
0928 16:55:53.927723 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:53.927819 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 16:55:53.928167 32601 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:41835: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
I20260928 16:55:53.953621 32526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } attempt: 5
Sep 28 16:55:53 dist-test-slave-wzcb krb5kdc[32446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614553, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for kudu/127.9.93.61@KRBTEST.COM
I20260928 16:55:53.968041 32608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.954703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49630 (local address 127.9.93.61:42991)
0928 16:55:53.954913 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:53.954919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:53.954938 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:55:53.955013 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:53.955017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:53.955082 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:53.955169 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:53.956078 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:53.956813 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:53.958054 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:53.958523 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:53.958645 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:53.962298 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:53.962303 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:53.962307 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:53.962345 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:53.965166 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:53.965939 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:53.965944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:53.965946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:53.966091 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:53.966491 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:53.966495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:53.966497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:53.966811 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:53.966976 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:53.967468 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 16:55:53.967702 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:55:53.968050 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.954452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:42991 (local address 127.0.0.1:49630)
0928 16:55:53.954646 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:53.954668 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:53.954814 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:53.955229 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:53.955234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:53.955705 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:53.955904 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:53.955914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:53.956983 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:53.956989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:53.957886 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:53.957901 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:53.958685 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:53.958691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:53.958917 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:53.959632 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:55:53.959651 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:53.962051 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:53.965349 (+ 3298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:53.965355 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:53.965360 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:53.965802 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:53.966226 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:53.966231 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:53.966233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:53.966376 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:53.966961 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:53.966966 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:53.967134 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 16:55:53.967275 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 16:55:53.967672 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 16:55:53.969564 32526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:55:53.970942 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.970435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41835 (local address 127.0.0.1:40738)
0928 16:55:53.970619 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:53.970724 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 16:55:53.972211 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.971785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41835 (local address 127.0.0.1:40744)
0928 16:55:53.971962 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:53.972061 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 16:55:53.972466 32526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:41835: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
I20260928 16:55:53.996393 32601 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } attempt: 2
I20260928 16:55:53.997612 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.997105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41835 (local address 127.0.0.1:40746)
0928 16:55:53.997318 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:53.997440 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 16:55:53.998879 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:53.998489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41835 (local address 127.0.0.1:40760)
0928 16:55:53.998635 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:53.998721 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 16:55:53.999115 32601 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:41835: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
I20260928 16:55:54.023741 32526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } attempt: 1
I20260928 16:55:54.025094 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.024582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41835 (local address 127.0.0.1:40776)
0928 16:55:54.024792 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:54.024907 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 16:55:54.026247 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.025843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41835 (local address 127.0.0.1:40784)
0928 16:55:54.026020 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:54.026104 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260928 16:55:54.026516 32526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:41835: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
I20260928 16:55:54.090358 32601 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } attempt: 3
I20260928 16:55:54.091790 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.091182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41835 (local address 127.0.0.1:40786)
0928 16:55:54.091496 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:54.091604 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260928 16:55:54.092975 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.092491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41835 (local address 127.0.0.1:40790)
0928 16:55:54.092742 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:54.092845 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260928 16:55:54.093194 32601 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:41835: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
I20260928 16:55:54.094712 32526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } attempt: 2
I20260928 16:55:54.095765 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.095266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41835 (local address 127.0.0.1:40794)
0928 16:55:54.095485 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:54.095584 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 16:55:54.096769 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.096414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41835 (local address 127.0.0.1:40798)
0928 16:55:54.096568 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:54.096647 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 16:55:54.096987 32526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:41835: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
I20260928 16:55:54.203250 32526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } attempt: 3
I20260928 16:55:54.204501 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.203980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41835 (local address 127.0.0.1:40802)
0928 16:55:54.204201 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:54.204308 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 16:55:54.205586 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.205190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41835 (local address 127.0.0.1:40804)
0928 16:55:54.205379 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:54.205458 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 16:55:54.205830 32526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:41835: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
I20260928 16:55:54.236479 32601 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } attempt: 4
I20260928 16:55:54.237723 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.237168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41835 (local address 127.0.0.1:40808)
0928 16:55:54.237443 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:54.237534 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 16:55:54.238859 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.238496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41835 (local address 127.0.0.1:40818)
0928 16:55:54.238659 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:54.238745 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 16:55:54.239105 32601 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.93.60:41835: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
W20260928 16:55:54.258908 32611 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:55:54.259294 32611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:55:54.259424 32611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:55:54.259519 32611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:55:54.267158 32611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:55:54.267295 32611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:55:54.267431 32611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:55:54.267500 32611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:55:54.267585 32611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:55:54.278280 32611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.93.20:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.93.62:39481,127.9.93.61:42991,127.9.93.60:41835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/krb5kdc/kudu_127.9.93.60.keytab
--principal=kudu/127.9.93.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.93.60:41835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.93.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:55:54.279899 32611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:55:54.281482 32611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:55:54.294642 32616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:54.294606 32617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:55:54.294675 32619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:55:54.295487 32611 server_base.cc:1034] running on GCE node
I20260928 16:55:54.296401 32611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:55:54.298133 32611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:55:54.299427 32611 hybrid_clock.cc:648] HybridClock initialized: now 1790614554299336 us; error 62 us; skew 500 ppm
I20260928 16:55:54.299805 32611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:55:54 dist-test-slave-wzcb krb5kdc[32446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614554, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:55:54.303790 32611 init.cc:377] Logged in from keytab as kudu/127.9.93.60@KRBTEST.COM (short username kudu)
I20260928 16:55:54.305552 32611 webserver.cc:460] Webserver started at http://127.9.93.60:40567/ using document root <none> and password file <none>
I20260928 16:55:54.306386 32611 fs_manager.cc:359] Metadata directory not provided
I20260928 16:55:54.306512 32611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:55:54.306869 32611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:55:54.311017 32611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-2/data/instance:
uuid: "2b37bf9e2ce14983b02ffdfeaa204bdb"
format_stamp: "Formatted at 2026-09-28 16:55:54 on dist-test-slave-wzcb"
server_key: "9960ec05479d6aa1cb1db7330be60b0b"
server_key_iv: "26db89172a08e2ec551bd730c71e8b87"
server_key_version: "encryptionkey@0"
I20260928 16:55:54.312108 32611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-2/wal/instance:
uuid: "2b37bf9e2ce14983b02ffdfeaa204bdb"
format_stamp: "Formatted at 2026-09-28 16:55:54 on dist-test-slave-wzcb"
server_key: "9960ec05479d6aa1cb1db7330be60b0b"
server_key_iv: "26db89172a08e2ec551bd730c71e8b87"
server_key_version: "encryptionkey@0"
I20260928 16:55:54.318301 32611 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:55:54.322659 32626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:55:54.324620 32611 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:55:54.324837 32611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-2/data,/tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-2/wal
uuid: "2b37bf9e2ce14983b02ffdfeaa204bdb"
format_stamp: "Formatted at 2026-09-28 16:55:54 on dist-test-slave-wzcb"
server_key: "9960ec05479d6aa1cb1db7330be60b0b"
server_key_iv: "26db89172a08e2ec551bd730c71e8b87"
server_key_version: "encryptionkey@0"
I20260928 16:55:54.325089 32611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:55:54.360442 32611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:55:54.365597 32611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:55:54.365926 32611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:55:54.383066 32526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } attempt: 4
I20260928 16:55:54.384475 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.383990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41835 (local address 127.0.0.1:40826)
0928 16:55:54.384200 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:54.384322 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.93.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 16:55:54.385717 32611 rpc_server.cc:307] RPC server started. Bound to: 127.9.93.60:41835
I20260928 16:55:54.385735 32678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.93.60:41835 every 8 connection(s)
I20260928 16:55:54.387456 32611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-2/data/info.pb
I20260928 16:55:54.392693 9588 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkzF10S/build/asan/bin/kudu as pid 32611
I20260928 16:55:54.392783 32680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:55:54.392923 9588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkzF10S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790614166338986-9588-0/minicluster-data/master-2/wal/instance
I20260928 16:55:54.393481 9588 external_mini_cluster.cc:1428] Setting key b34ac62f6db7408be1379d1921cc2121
Sep 28 16:55:54 dist-test-slave-wzcb krb5kdc[32446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614553, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.62@KRBTEST.COM for kudu/127.9.93.60@KRBTEST.COM
I20260928 16:55:54.405037 32680 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39481 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:55:54.411835 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.385499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41835 (local address 127.0.0.1:40828)
0928 16:55:54.385689 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:54.385711 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:54.385872 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:54.391824 (+ 5952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:54.391829 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:54.392388 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:54.392639 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:54.392658 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.395211 (+ 2553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.395217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:54.396601 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:54.396626 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.398245 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.398252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:54.398470 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:54.399241 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:55:54.399268 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:54.402283 (+ 3015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:54.407747 (+ 5464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:54.407754 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:54.407758 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:54.408368 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:54.408982 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:54.408987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:54.408991 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:54.409200 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:54.410364 (+ 1164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:54.410370 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:54.410610 (+ 240us) client_negotiation.cc:769] Sending connection context
0928 16:55:54.411015 (+ 405us) client_negotiation.cc:240] Negotiation successful
0928 16:55:54.411415 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
Sep 28 16:55:54 dist-test-slave-wzcb krb5kdc[32446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.60@KRBTEST.COM
I20260928 16:55:54.415133 32679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.387122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40828 (local address 127.9.93.60:41835)
0928 16:55:54.388009 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:54.388024 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:54.388092 (+ 68us) server_negotiation.cc:407] Connection header received
0928 16:55:54.390590 (+ 2498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:54.390624 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:54.391243 (+ 619us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:54.391799 (+ 556us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:54.392881 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.394990 (+ 2109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.396840 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.398018 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.402612 (+ 4594us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:54.403060 (+ 448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:54.403074 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:54.403098 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:54.403144 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:54.407571 (+ 4427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:54.408561 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:54.408571 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:54.408585 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:54.408824 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:54.409363 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:54.409367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:54.409371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:54.410185 (+ 814us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:54.412478 (+ 2293us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:54.412848 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 16:55:54.413444 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":640,"thread_start_us":418,"threads_started":1}
I20260928 16:55:54.423749 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.394515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40842 (local address 127.9.93.60:41835)
0928 16:55:54.404318 (+ 9803us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:54.404325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:54.404356 (+ 31us) server_negotiation.cc:407] Connection header received
0928 16:55:54.404407 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:54.404410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:54.404473 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:54.404582 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:54.406318 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.407345 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.409094 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.409823 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.409936 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:54.415744 (+ 5808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:54.415751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:54.415756 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:54.415810 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:54.419232 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:54.420314 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:54.420320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:54.420323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:54.420512 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:54.421075 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:54.421080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:54.421084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:54.421462 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:54.421750 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:54.423023 (+ 1273us) server_negotiation.cc:299] Negotiation successful
0928 16:55:54.423292 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":9618,"thread_start_us":9378,"threads_started":1}
I20260928 16:55:54.425462 32526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02: Bootstrap starting.
I20260928 16:55:54.430904 32526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:54.432647 32526 log.cc:824] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:55:54 dist-test-slave-wzcb krb5kdc[32446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614554, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.60@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
Sep 28 16:55:54 dist-test-slave-wzcb krb5kdc[32446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.62@KRBTEST.COM
I20260928 16:55:54.437430 32526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02: No bootstrap required, opened a new log
I20260928 16:55:54.443508 32684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.408612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44198 (local address 127.9.93.62:39481)
0928 16:55:54.426609 (+ 17997us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:54.426616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:54.426636 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:55:54.426707 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:54.426713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:54.426804 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:54.426943 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:54.428637 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.429969 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.432004 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.432606 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.432743 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:54.436723 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:54.436728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:54.436732 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:54.436772 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:54.440180 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:54.441026 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:54.441030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:54.441033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:54.441184 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:54.441621 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:54.441625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:54.441627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:54.441978 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:54.442142 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:54.442748 (+ 606us) server_negotiation.cc:299] Negotiation successful
0928 16:55:54.443012 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":17842,"thread_start_us":18479,"threads_started":1}
I20260928 16:55:54.443497 32685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.408655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.62:39481 (local address 127.0.0.1:44198)
0928 16:55:54.424573 (+ 15918us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:54.424629 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:54.424786 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:54.427031 (+ 2245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:54.427040 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:54.428047 (+ 1007us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:54.428382 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:54.428404 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.430157 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.430163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:54.431834 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:54.431849 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.432827 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.432835 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:54.433067 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:54.433797 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:55:54.433826 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:54.436459 (+ 2633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:54.440352 (+ 3893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:54.440372 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:54.440381 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:54.440879 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:54.441313 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:54.441319 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:54.441323 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:54.441490 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:54.442133 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:54.442143 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:54.442415 (+ 272us) client_negotiation.cc:769] Sending connection context
0928 16:55:54.442602 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 16:55:54.443009 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15636,"thread_start_us":14671,"threads_started":1}
I20260928 16:55:54.445420 32680 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:55:54.446221 32686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.425950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44204 (local address 127.9.93.62:39481)
0928 16:55:54.426659 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:54.426664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:54.426682 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:55:54.426715 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:54.426719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:54.426833 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:54.426943 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:54.428574 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.429592 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.431439 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.432158 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.432308 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:54.437547 (+ 5239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:54.437554 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:54.437558 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:54.437600 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:54.441603 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:54.442700 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:54.442705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:54.442709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:54.442926 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:54.443457 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:54.443462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:54.443465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:54.443870 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:54.444172 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:54.445466 (+ 1294us) server_negotiation.cc:299] Negotiation successful
0928 16:55:54.445735 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":609,"spinlock_wait_cycles":57088,"thread_start_us":337,"threads_started":1}
I20260928 16:55:54.450026 32526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19952321790f40119b0dc29ac875fb02" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39481 } } peers { permanent_uuid: "4df781a8a68547038f3688651fbaa9ae" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } } peers { permanent_uuid: "2b37bf9e2ce14983b02ffdfeaa204bdb" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } }
I20260928 16:55:54.450544 32526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19952321790f40119b0dc29ac875fb02, State: Initialized, Role: FOLLOWER
I20260928 16:55:54.451222 32526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19952321790f40119b0dc29ac875fb02" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39481 } } peers { permanent_uuid: "4df781a8a68547038f3688651fbaa9ae" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } } peers { permanent_uuid: "2b37bf9e2ce14983b02ffdfeaa204bdb" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } }
I20260928 16:55:54.452524 32690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19952321790f40119b0dc29ac875fb02" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39481 } } peers { permanent_uuid: "4df781a8a68547038f3688651fbaa9ae" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } } peers { permanent_uuid: "2b37bf9e2ce14983b02ffdfeaa204bdb" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } } }
I20260928 16:55:54.453104 32690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02 [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 16:55:54 dist-test-slave-wzcb krb5kdc[32446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614554, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.60@KRBTEST.COM for kudu/127.9.93.61@KRBTEST.COM
I20260928 16:55:54.454281 32526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 16:55:54 dist-test-slave-wzcb krb5kdc[32446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.93.61@KRBTEST.COM
I20260928 16:55:54.462599 32685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.446170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.61:42991 (local address 127.0.0.1:49640)
0928 16:55:54.446602 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:54.446619 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:54.446735 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:54.447078 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:54.447083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:54.447600 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:54.447767 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:54.447778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.448958 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.448964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:54.450073 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:54.450089 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.451087 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.451094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:54.451283 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:54.452115 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:55:54.452140 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:54.454891 (+ 2751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:54.459294 (+ 4403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:54.459326 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:54.459331 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:54.459941 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:54.460543 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:54.460549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:54.460553 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:54.460739 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:54.461496 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:54.461502 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:54.461693 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 16:55:54.461904 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 16:55:54.462216 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309}
I20260928 16:55:54.463178 32608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.446244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49640 (local address 127.9.93.61:42991)
0928 16:55:54.446519 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:54.446525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:54.446687 (+ 162us) server_negotiation.cc:407] Connection header received
0928 16:55:54.446856 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:54.446862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:54.446951 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:54.447038 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:54.447929 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.448814 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.450265 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.450896 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.451025 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:54.455135 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:54.455143 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:54.455148 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:54.455195 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:54.459076 (+ 3881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:54.460093 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:54.460099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:54.460103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:54.460355 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:54.460911 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:54.460917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:54.460921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:54.461343 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:54.461516 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:54.462368 (+ 852us) server_negotiation.cc:299] Negotiation successful
0928 16:55:54.462717 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260928 16:55:54.464450 32680 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:55:54.466986 32689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.449048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49656 (local address 127.9.93.61:42991)
0928 16:55:54.449660 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:54.449666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:54.449689 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:55:54.449738 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:54.449743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:54.449835 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:54.449944 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:54.451754 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.452892 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.454448 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.455116 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.455234 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:54.460140 (+ 4906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:54.460148 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:54.460153 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:54.460222 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:54.463919 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:54.464753 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:54.464757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:54.464759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:54.464897 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:54.465339 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:54.465344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:54.465348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:54.465730 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:54.466001 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:54.466318 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 16:55:54.466566 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":260,"threads_started":1}
W20260928 16:55:54.471448 32480 tablet.cc:2369] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:55:54.472648 9588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Sep 28 16:55:54 dist-test-slave-wzcb krb5kdc[32446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614554, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.60@KRBTEST.COM for kudu/127.9.93.60@KRBTEST.COM
I20260928 16:55:54.475768 32501 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "19952321790f40119b0dc29ac875fb02"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44204
I20260928 16:55:54.476300 32501 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 16:55:54.476534 32501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02 [term 0 FOLLOWER]: Advancing to term 1
W20260928 16:55:54.480172 32701 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:55:54.480463 32701 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:55:54.481571 32501 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19952321790f40119b0dc29ac875fb02" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39481 } } peers { permanent_uuid: "4df781a8a68547038f3688651fbaa9ae" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } } peers { permanent_uuid: "2b37bf9e2ce14983b02ffdfeaa204bdb" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } }
I20260928 16:55:54.483137 32501 leader_election.cc:288] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02 [CANDIDATE]: Term 1 election: Requested vote from peers 4df781a8a68547038f3688651fbaa9ae (127.9.93.61:42991), 2b37bf9e2ce14983b02ffdfeaa204bdb (127.9.93.60:41835)
I20260928 16:55:54.474864 32459 master_runner.cc:186] Error getting master registration for 127.9.93.61:42991: OK, instance_id { permanent_uuid: "4df781a8a68547038f3688651fbaa9ae" instance_seqno: 1790614553801983 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:55:54.486598 32459 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:55:54.488655 32685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.465298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41835 (local address 127.0.0.1:40846)
0928 16:55:54.465504 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:54.465526 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:54.465632 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:54.466189 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:54.466194 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:54.466756 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:54.467000 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:54.467015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.468442 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.468448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:54.469835 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:54.469853 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.470938 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.470944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:54.471147 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:54.471859 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:55:54.471884 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:54.474541 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:54.485401 (+ 10860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:54.485406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:54.485410 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:54.485981 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:54.486435 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:54.486440 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:54.486444 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:54.486601 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:54.487431 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:54.487438 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:54.487629 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 16:55:54.487831 (+ 202us) client_negotiation.cc:240] Negotiation successful
0928 16:55:54.488228 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 16:55:54.489881 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.465540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40846 (local address 127.9.93.60:41835)
0928 16:55:54.465737 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:54.465743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:54.465763 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:55:54.465823 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:54.465829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:54.465911 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:54.466012 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:54.467165 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.468272 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.469990 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.470590 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.470689 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:54.474740 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:54.474747 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:54.474753 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:54.474823 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:54.485240 (+ 10417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:54.486145 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:54.486150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:54.486154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:54.486312 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:54.486735 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:54.486740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:54.486743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:54.487231 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:54.487444 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:54.489204 (+ 1760us) server_negotiation.cc:299] Negotiation successful
0928 16:55:54.489490 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:55:54.497010 32680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b37bf9e2ce14983b02ffdfeaa204bdb: Bootstrap starting.
I20260928 16:55:54.504985 32680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b37bf9e2ce14983b02ffdfeaa204bdb: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:54.508821 32680 log.cc:824] T 00000000000000000000000000000000 P 2b37bf9e2ce14983b02ffdfeaa204bdb: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:54.526284 32680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b37bf9e2ce14983b02ffdfeaa204bdb: No bootstrap required, opened a new log
I20260928 16:55:54.538342 32601 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } attempt: 5
I20260928 16:55:54.538755 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.506130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40854 (local address 127.9.93.60:41835)
0928 16:55:54.506322 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:54.506329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:54.512182 (+ 5853us) server_negotiation.cc:407] Connection header received
0928 16:55:54.512704 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:54.512711 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:54.512803 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:54.512876 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:54.517701 (+ 4825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.518812 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.520575 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.521281 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.521395 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:54.525164 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:54.525172 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:54.525178 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:54.525239 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:54.528948 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:54.531552 (+ 2604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:54.531559 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:54.531563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:54.532021 (+ 458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:54.533776 (+ 1755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:54.533783 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:54.533787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:54.534887 (+ 1100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:54.535551 (+ 664us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:54.537269 (+ 1718us) server_negotiation.cc:299] Negotiation successful
0928 16:55:54.537872 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:55:54.544389 32689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.494368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49668 (local address 127.9.93.61:42991)
0928 16:55:54.494629 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:54.494636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:54.507545 (+ 12909us) server_negotiation.cc:407] Connection header received
0928 16:55:54.507693 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:54.507700 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:54.507804 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:54.507905 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:54.510315 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.511517 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.513470 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.514147 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.514277 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:54.533395 (+ 19118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:54.533404 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:54.533409 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:54.533464 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:54.537338 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:54.539080 (+ 1742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:54.539087 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:54.539090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:54.539279 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:54.540553 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:54.540559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:54.540562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:54.540983 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:54.541243 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:54.543769 (+ 2526us) server_negotiation.cc:299] Negotiation successful
0928 16:55:54.544038 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 16:55:54.545397 32686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.494794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44208 (local address 127.9.93.62:39481)
0928 16:55:54.495053 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:54.495059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:54.505330 (+ 10271us) server_negotiation.cc:407] Connection header received
0928 16:55:54.507814 (+ 2484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:54.507820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:54.507920 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:54.507992 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:54.515052 (+ 7060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.516144 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.517541 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.518171 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.518326 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:54.523467 (+ 5141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:54.523475 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:54.523481 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:54.523534 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:54.531059 (+ 7525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:54.539856 (+ 8797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:54.539862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:54.539865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:54.540065 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:54.541785 (+ 1720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:54.541792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:54.541795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:54.542260 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:54.542573 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:54.543447 (+ 874us) server_negotiation.cc:299] Negotiation successful
0928 16:55:54.544154 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 16:55:54.548303 32680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b37bf9e2ce14983b02ffdfeaa204bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19952321790f40119b0dc29ac875fb02" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39481 } } peers { permanent_uuid: "4df781a8a68547038f3688651fbaa9ae" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } } peers { permanent_uuid: "2b37bf9e2ce14983b02ffdfeaa204bdb" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } }
I20260928 16:55:54.549003 32680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b37bf9e2ce14983b02ffdfeaa204bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b37bf9e2ce14983b02ffdfeaa204bdb, State: Initialized, Role: FOLLOWER
Sep 28 16:55:54 dist-test-slave-wzcb krb5kdc[32446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614553, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.93.61@KRBTEST.COM for kudu/127.9.93.60@KRBTEST.COM
I20260928 16:55:54.549701 32680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b37bf9e2ce14983b02ffdfeaa204bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19952321790f40119b0dc29ac875fb02" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39481 } } peers { permanent_uuid: "4df781a8a68547038f3688651fbaa9ae" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } } peers { permanent_uuid: "2b37bf9e2ce14983b02ffdfeaa204bdb" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } }
I20260928 16:55:54.554071 32680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b37bf9e2ce14983b02ffdfeaa204bdb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:55:54.558347 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.539070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.93.60:41835 (local address 127.0.0.1:40866)
0928 16:55:54.539581 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 16:55:54.539603 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:55:54.539714 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:55:54.540900 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:55:54.540906 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:55:54.541451 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:55:54.541683 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:54.541696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.543330 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.543338 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:54.544941 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:55:54.544962 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.546710 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.546718 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:55:54.546978 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:54.547825 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:55:54.547882 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:55:54.550776 (+ 2894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:55:54.555114 (+ 4338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:54.555122 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:54.555126 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:54.555775 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:54.556258 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:55:54.556263 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:55:54.556266 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:55:54.556467 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:55:54.557282 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:55:54.557287 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:55:54.557466 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 16:55:54.557601 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 16:55:54.557935 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368}
I20260928 16:55:54.558341 32653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19952321790f40119b0dc29ac875fb02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2b37bf9e2ce14983b02ffdfeaa204bdb"
I20260928 16:55:54.558976 32653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b37bf9e2ce14983b02ffdfeaa204bdb [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:55:54.559463 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:54.539946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40866 (local address 127.9.93.60:41835)
0928 16:55:54.540418 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:54.540424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:54.540555 (+ 131us) server_negotiation.cc:407] Connection header received
0928 16:55:54.540622 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:54.540627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:54.540711 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:54.541847 (+ 1136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:54.541991 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.543130 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.545483 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:54.546505 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:54.547371 (+ 866us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:54.551046 (+ 3675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:55:54.551054 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:55:54.551059 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:55:54.551105 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:55:54.554867 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:54.555946 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:54.555952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:54.555956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:54.556113 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:55:54.556610 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:55:54.556615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:55:54.556618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:55:54.557013 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:55:54.557198 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:54.558791 (+ 1593us) server_negotiation.cc:299] Negotiation successful
0928 16:55:54.559077 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347}
I20260928 16:55:54.565797 32601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4df781a8a68547038f3688651fbaa9ae: Bootstrap starting.
I20260928 16:55:54.568311 32653 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2b37bf9e2ce14983b02ffdfeaa204bdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19952321790f40119b0dc29ac875fb02 in term 1.
I20260928 16:55:54.569535 32477 leader_election.cc:302] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19952321790f40119b0dc29ac875fb02, 2b37bf9e2ce14983b02ffdfeaa204bdb; no voters:
I20260928 16:55:54.570178 32690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:55:54.571043 32690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02 [term 1 LEADER]: Becoming Leader. State: Replica: 19952321790f40119b0dc29ac875fb02, State: Running, Role: LEADER
I20260928 16:55:54.571624 32711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b37bf9e2ce14983b02ffdfeaa204bdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19952321790f40119b0dc29ac875fb02" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39481 } } peers { permanent_uuid: "4df781a8a68547038f3688651fbaa9ae" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } } peers { permanent_uuid: "2b37bf9e2ce14983b02ffdfeaa204bdb" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } } }
I20260928 16:55:54.571777 32690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19952321790f40119b0dc29ac875fb02" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39481 } } peers { permanent_uuid: "4df781a8a68547038f3688651fbaa9ae" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } } peers { permanent_uuid: "2b37bf9e2ce14983b02ffdfeaa204bdb" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } }
I20260928 16:55:54.572047 32711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b37bf9e2ce14983b02ffdfeaa204bdb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:55:54.572921 32601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4df781a8a68547038f3688651fbaa9ae: Neither blocks nor log segments found. Creating new log.
I20260928 16:55:54.574528 32601 log.cc:824] T 00000000000000000000000000000000 P 4df781a8a68547038f3688651fbaa9ae: Log is configured to *not* fsync() on all Append() calls
I20260928 16:55:54.585156 32601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4df781a8a68547038f3688651fbaa9ae: No bootstrap required, opened a new log
I20260928 16:55:54.594517 32713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19952321790f40119b0dc29ac875fb02. Latest consensus state: current_term: 1 leader_uuid: "19952321790f40119b0dc29ac875fb02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19952321790f40119b0dc29ac875fb02" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39481 } } peers { permanent_uuid: "4df781a8a68547038f3688651fbaa9ae" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } } peers { permanent_uuid: "2b37bf9e2ce14983b02ffdfeaa204bdb" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } } }
I20260928 16:55:54.595423 32713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02 [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:54.596999 32601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4df781a8a68547038f3688651fbaa9ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19952321790f40119b0dc29ac875fb02" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39481 } } peers { permanent_uuid: "4df781a8a68547038f3688651fbaa9ae" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } } peers { permanent_uuid: "2b37bf9e2ce14983b02ffdfeaa204bdb" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } }
I20260928 16:55:54.597532 32601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4df781a8a68547038f3688651fbaa9ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df781a8a68547038f3688651fbaa9ae, State: Initialized, Role: FOLLOWER
I20260928 16:55:54.598184 32601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4df781a8a68547038f3688651fbaa9ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19952321790f40119b0dc29ac875fb02" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39481 } } peers { permanent_uuid: "4df781a8a68547038f3688651fbaa9ae" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } } peers { permanent_uuid: "2b37bf9e2ce14983b02ffdfeaa204bdb" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } }
W20260928 16:55:54.618544 32632 tablet.cc:2369] T 00000000000000000000000000000000 P 2b37bf9e2ce14983b02ffdfeaa204bdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:55:54.619207 32724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:55:54.624588 32601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4df781a8a68547038f3688651fbaa9ae [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:55:54.625161 32576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19952321790f40119b0dc29ac875fb02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4df781a8a68547038f3688651fbaa9ae"
I20260928 16:55:54.625774 32576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4df781a8a68547038f3688651fbaa9ae [term 0 FOLLOWER]: Advancing to term 1
W20260928 16:55:54.632707 32729 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2b37bf9e2ce14983b02ffdfeaa204bdb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:55:54.632886 32729 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:55:54.635990 32576 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4df781a8a68547038f3688651fbaa9ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19952321790f40119b0dc29ac875fb02 in term 1.
I20260928 16:55:54.637924 32725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4df781a8a68547038f3688651fbaa9ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19952321790f40119b0dc29ac875fb02" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39481 } } peers { permanent_uuid: "4df781a8a68547038f3688651fbaa9ae" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } } peers { permanent_uuid: "2b37bf9e2ce14983b02ffdfeaa204bdb" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } } }
I20260928 16:55:54.640651 32725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4df781a8a68547038f3688651fbaa9ae [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:55:54.661209 32724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:55:54.647429 32611 master_runner.cc:186] Error getting master registration for 127.9.93.61:42991: OK, instance_id { permanent_uuid: "4df781a8a68547038f3688651fbaa9ae" instance_seqno: 1790614553801983 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:55:54.677985 32611 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:55:54.685890 32653 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2b37bf9e2ce14983b02ffdfeaa204bdb [term 1 FOLLOWER]: Refusing update from remote peer 19952321790f40119b0dc29ac875fb02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:55:54.687806 32713 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b37bf9e2ce14983b02ffdfeaa204bdb" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:55:54.702821 32576 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4df781a8a68547038f3688651fbaa9ae [term 1 FOLLOWER]: Refusing update from remote peer 19952321790f40119b0dc29ac875fb02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:55:54.704339 32713 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4df781a8a68547038f3688651fbaa9ae" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:55:54.744850 32725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4df781a8a68547038f3688651fbaa9ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19952321790f40119b0dc29ac875fb02. Latest consensus state: current_term: 1 leader_uuid: "19952321790f40119b0dc29ac875fb02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19952321790f40119b0dc29ac875fb02" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39481 } } peers { permanent_uuid: "4df781a8a68547038f3688651fbaa9ae" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } } peers { permanent_uuid: "2b37bf9e2ce14983b02ffdfeaa204bdb" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } } }
I20260928 16:55:54.745291 32725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4df781a8a68547038f3688651fbaa9ae [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:55:54.748368 32732 mvcc.cc:205] Tried to move back new op lower bound from 7334357215975804928 to 7334357215533355008. Current Snapshot: MvccSnapshot[applied={T|T < 7334357215975804928}]
I20260928 16:55:54.750666 32746 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 16:55:54.753130 32711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b37bf9e2ce14983b02ffdfeaa204bdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19952321790f40119b0dc29ac875fb02. Latest consensus state: current_term: 1 leader_uuid: "19952321790f40119b0dc29ac875fb02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19952321790f40119b0dc29ac875fb02" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39481 } } peers { permanent_uuid: "4df781a8a68547038f3688651fbaa9ae" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } } peers { permanent_uuid: "2b37bf9e2ce14983b02ffdfeaa204bdb" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } } }
I20260928 16:55:54.753435 32711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b37bf9e2ce14983b02ffdfeaa204bdb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:55:54.757109 32711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b37bf9e2ce14983b02ffdfeaa204bdb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "19952321790f40119b0dc29ac875fb02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19952321790f40119b0dc29ac875fb02" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39481 } } peers { permanent_uuid: "4df781a8a68547038f3688651fbaa9ae" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } } peers { permanent_uuid: "2b37bf9e2ce14983b02ffdfeaa204bdb" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } } }
I20260928 16:55:54.757423 32711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b37bf9e2ce14983b02ffdfeaa204bdb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:55:54.760012 32739 mvcc.cc:205] Tried to move back new op lower bound from 7334357215975804928 to 7334357215533355008. Current Snapshot: MvccSnapshot[applied={T|T < 7334357215975804928}]
I20260928 16:55:54.763875 32713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "19952321790f40119b0dc29ac875fb02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19952321790f40119b0dc29ac875fb02" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39481 } } peers { permanent_uuid: "4df781a8a68547038f3688651fbaa9ae" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } } peers { permanent_uuid: "2b37bf9e2ce14983b02ffdfeaa204bdb" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } } }
I20260928 16:55:54.764223 32713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02 [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:54.765271 32725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4df781a8a68547038f3688651fbaa9ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "19952321790f40119b0dc29ac875fb02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19952321790f40119b0dc29ac875fb02" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39481 } } peers { permanent_uuid: "4df781a8a68547038f3688651fbaa9ae" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } } peers { permanent_uuid: "2b37bf9e2ce14983b02ffdfeaa204bdb" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } } }
I20260928 16:55:54.765565 32725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4df781a8a68547038f3688651fbaa9ae [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:55:54.768162 32724 catalog_manager.cc:1347] Generated new cluster ID: 92cea89c0063468784af49f9c6054a7c
I20260928 16:55:54.768285 32724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:55:54.776556 32690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "19952321790f40119b0dc29ac875fb02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19952321790f40119b0dc29ac875fb02" member_type: VOTER last_known_addr { host: "127.9.93.62" port: 39481 } } peers { permanent_uuid: "4df781a8a68547038f3688651fbaa9ae" member_type: VOTER last_known_addr { host: "127.9.93.61" port: 42991 } } peers { permanent_uuid: "2b37bf9e2ce14983b02ffdfeaa204bdb" member_type: VOTER last_known_addr { host: "127.9.93.60" port: 41835 } } }
I20260928 16:55:54.777160 32690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02 [sys.catalog]: This master's current role is: LEADER
I20260928 16:55:54.784498 32724 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:55:54.789472 32724 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:55:54.823592 32724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19952321790f40119b0dc29ac875fb02: Generated new TSK 0
I20260928 16:55:54.825073 32724 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:55:54.930982 32686 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:55:54.916228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.9.93.62:39481)
0928 16:55:54.916432 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:54.916440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:54.928389 (+ 11949us) server_negotiation.cc:407] Connection header received
0928 16:55:54.929040 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:54.929046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:54.929154 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:54.929242 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:54.930642 (+ 1400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44218: BlockingRecv error: recv got EOF from 127.0.0.1:44218 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 16:55:54.932199 32689 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:55:54.916407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49674 (local address 127.9.93.61:42991)
0928 16:55:54.916601 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:54.916607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:54.928574 (+ 11967us) server_negotiation.cc:407] Connection header received
0928 16:55:54.930130 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:54.930137 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:54.930249 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:54.930329 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:54.931920 (+ 1591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49674: BlockingRecv error: recv got EOF from 127.0.0.1:49674 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 16:55:54.933846 32682 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:55:54.929246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40868 (local address 127.9.93.60:41835)
0928 16:55:54.929452 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:54.929459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:54.932087 (+ 2628us) server_negotiation.cc:407] Connection header received
0928 16:55:54.932339 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:54.932345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:54.932440 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:54.932515 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:54.933566 (+ 1051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40868: BlockingRecv error: recv got EOF from 127.0.0.1:40868 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 16:55:54.942061 32689 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:55:54.939543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49678 (local address 127.9.93.61:42991)
0928 16:55:54.939741 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:54.939747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:54.940288 (+ 541us) server_negotiation.cc:407] Connection header received
0928 16:55:54.940567 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:54.940575 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:54.940768 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:54.940896 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:54.941813 (+ 917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49678: BlockingRecv error: recv got EOF from 127.0.0.1:49678 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 16:55:54.949743 32689 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:55:54.946839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49694 (local address 127.9.93.61:42991)
0928 16:55:54.947051 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:54.947057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:54.947623 (+ 566us) server_negotiation.cc:407] Connection header received
0928 16:55:54.947827 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:54.947834 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:54.947951 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:54.948120 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:54.949455 (+ 1335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49694: BlockingRecv error: recv got EOF from 127.0.0.1:49694 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260928 16:55:54.959417 32689 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:55:54.956835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49700 (local address 127.9.93.61:42991)
0928 16:55:54.957038 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:54.957044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:54.957620 (+ 576us) server_negotiation.cc:407] Connection header received
0928 16:55:54.957871 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:54.957878 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:54.957999 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:54.958134 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:54.959074 (+ 940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49700: BlockingRecv error: recv got EOF from 127.0.0.1:49700 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 16:55:54.972810 32689 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:55:54.969853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49708 (local address 127.9.93.61:42991)
0928 16:55:54.970120 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:54.970128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:54.970695 (+ 567us) server_negotiation.cc:407] Connection header received
0928 16:55:54.971086 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:54.971093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:54.971207 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:54.971293 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:54.972531 (+ 1238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49708: BlockingRecv error: recv got EOF from 127.0.0.1:49708 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
W20260928 16:55:54.989737 32689 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:55:54.987119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49716 (local address 127.9.93.61:42991)
0928 16:55:54.987349 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:54.987356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:54.987964 (+ 608us) server_negotiation.cc:407] Connection header received
0928 16:55:54.988300 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:54.988307 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:54.988430 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:54.988536 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:54.989490 (+ 954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49716: BlockingRecv error: recv got EOF from 127.0.0.1:49716 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 16:55:55.012598 32689 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:55:55.009954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49718 (local address 127.9.93.61:42991)
0928 16:55:55.010206 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:55.010212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:55.010864 (+ 652us) server_negotiation.cc:407] Connection header received
0928 16:55:55.011151 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:55.011157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:55.011262 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:55.011417 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:55.012342 (+ 925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49718: BlockingRecv error: recv got EOF from 127.0.0.1:49718 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
W20260928 16:55:55.051594 32689 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:55:55.049051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49720 (local address 127.9.93.61:42991)
0928 16:55:55.049300 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:55.049307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:55.049728 (+ 421us) server_negotiation.cc:407] Connection header received
0928 16:55:55.050018 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:55.050023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:55.050122 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:55.050226 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:55.051273 (+ 1047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49720: BlockingRecv error: recv got EOF from 127.0.0.1:49720 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260928 16:55:55.122605 32689 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:55:55.120046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49730 (local address 127.9.93.61:42991)
0928 16:55:55.120287 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:55.120295 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:55.120937 (+ 642us) server_negotiation.cc:407] Connection header received
0928 16:55:55.121140 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:55.121145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:55.121234 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:55.121338 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:55.122362 (+ 1024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49730: BlockingRecv error: recv got EOF from 127.0.0.1:49730 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 16:55:55.257117 32689 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:55:55.254847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49732 (local address 127.9.93.61:42991)
0928 16:55:55.255066 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:55.255074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:55.255645 (+ 571us) server_negotiation.cc:407] Connection header received
0928 16:55:55.255876 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:55.255880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:55.255979 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:55.256082 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:55.256929 (+ 847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49732: BlockingRecv error: recv got EOF from 127.0.0.1:49732 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 16:55:55.527750 32689 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:55:55.525463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49748 (local address 127.9.93.61:42991)
0928 16:55:55.525751 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:55.525759 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:55.526039 (+ 280us) server_negotiation.cc:407] Connection header received
0928 16:55:55.526349 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:55.526354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:55.526446 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:55.526538 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:55:55.527444 (+ 906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49748: BlockingRecv error: recv got EOF from 127.0.0.1:49748 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 16:55:55.635152 32729 catalog_manager.cc:1259] Loaded cluster ID: 92cea89c0063468784af49f9c6054a7c
I20260928 16:55:55.635291 32729 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2b37bf9e2ce14983b02ffdfeaa204bdb: loading cluster ID for follower catalog manager: success
I20260928 16:55:55.638217 32729 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2b37bf9e2ce14983b02ffdfeaa204bdb: acquiring CA information for follower catalog manager: success
I20260928 16:55:55.640367 32729 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2b37bf9e2ce14983b02ffdfeaa204bdb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:55:55.756230 32746 catalog_manager.cc:1259] Loaded cluster ID: 92cea89c0063468784af49f9c6054a7c
I20260928 16:55:55.756376 32746 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4df781a8a68547038f3688651fbaa9ae: loading cluster ID for follower catalog manager: success
I20260928 16:55:55.759559 32746 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4df781a8a68547038f3688651fbaa9ae: acquiring CA information for follower catalog manager: success
I20260928 16:55:55.762032 32746 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4df781a8a68547038f3688651fbaa9ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:55:56.065038 349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:56.056730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49758 (local address 127.9.93.61:42991)
0928 16:55:56.057615 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:56.057620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:56.057642 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:55:56.057715 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:56.057719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:56.057928 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:56.058045 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:55:56.058734 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:56.059751 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:56.061895 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:56.062482 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:56.062608 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:56.063421 (+ 813us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:55:56.064164 (+ 743us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:55:56.064304 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:56.064494 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 16:55:56.064587 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":573,"threads_started":1}
I20260928 16:55:56.082856 355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:56.075885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40870 (local address 127.9.93.60:41835)
0928 16:55:56.076999 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:56.077004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:56.077022 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:55:56.077131 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:56.077134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:56.077318 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:56.077495 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:55:56.078052 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:56.078842 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:56.080354 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:56.080781 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:56.080908 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:56.081515 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:55:56.082151 (+ 636us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:55:56.082323 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:56.082430 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 16:55:56.082534 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":999,"thread_start_us":877,"threads_started":1}
I20260928 16:55:56.102038 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:56.089887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44226 (local address 127.9.93.62:39481)
0928 16:55:56.093554 (+ 3667us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:56.093559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:56.093579 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:55:56.093699 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:56.093704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:56.094138 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:56.094348 (+ 210us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:55:56.095135 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:56.096301 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:56.098690 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:56.099402 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:56.099544 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:56.100325 (+ 781us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:55:56.100873 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:55:56.100969 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:56.101589 (+ 620us) server_negotiation.cc:299] Negotiation successful
0928 16:55:56.101695 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3204,"thread_start_us":3086,"threads_started":1}
I20260928 16:55:56.102342 349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:56.090210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49770 (local address 127.9.93.61:42991)
0928 16:55:56.090428 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:56.090433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:56.095179 (+ 4746us) server_negotiation.cc:407] Connection header received
0928 16:55:56.095546 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:56.095553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:56.095660 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:56.095729 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:55:56.096562 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:56.097727 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:56.099950 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:56.100659 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:56.100770 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:56.101493 (+ 723us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:55:56.101643 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:55:56.101709 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:56.101953 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 16:55:56.102056 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 16:55:56.107626 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 32459
I20260928 16:55:56.108983 32521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:55:56.110069 355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:55:56.095488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40874 (local address 127.9.93.60:41835)
0928 16:55:56.095671 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 16:55:56.095677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:55:56.103351 (+ 7674us) server_negotiation.cc:407] Connection header received
0928 16:55:56.103719 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:55:56.103726 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:55:56.103827 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:55:56.103908 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:55:56.104517 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:56.105609 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:56.107627 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:55:56.108256 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:55:56.108367 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:55:56.109114 (+ 747us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:55:56.109286 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:55:56.109393 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 16:55:56.109629 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 16:55:56.109759 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 16:55:56.244572 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 32459
I20260928 16:55:56.265937 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 32533
I20260928 16:55:56.267354 32596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:55:56.380048 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 32533
I20260928 16:55:56.399178 9588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkzF10S/build/asan/bin/kudu pid 32611
I20260928 16:55:56.400552 32673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:55:56.512840 9588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkzF10S/build/asan/bin/kudu with pid 32611
2026-09-28T16:55:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5165 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9989 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (390133 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST